Check and identify Lin Files by contents

[sajjansinghania]

I have over 300 *.lin files looking as under. Everytime when an application is run, it creates more files. My need is to check contents of each file and to identify 100% duplicate files for deletion at weekly interval. Kindly help.

Some:
qx|o1|md|3SJT64HQDCJ9765432,SQ87HT6DQJT85432C,SK9H874DAK96CKQT8|rh||ah|Board 1|sv|0|pg||
qx|o2|md|4SAKTHJDKJT987542C,S6542HA62DA3CQ743,SQ83HK8DQCAJT9852|rh||ah|Board 2|sv|n|pg||
qx|o3|md|1S7H8DKQT9753CAK93,SHAKQJT543DAJ82C2,SAK984H96D6CQJT86|rh||ah|Board 3|sv|e|pg||
qx|o4|md|2S76H6DAQT987432C5,SQT3HT987532D5C62,SA42HAQJDCQJT9843|rh||ah|Board 4|sv|b|pg||
qx|o5|md|3S874H42D54CQJT872,SJ9HDAQJ98732C543,SK5HAJ97653DCAK96|rh||ah|Board 5|sv|n|pg||
qx|o6|md|4SAKQ854H8DQJ6C742,S93HDAT9542CAKJ85,SJT762H64DK7CQT96|rh||ah|Board 6|sv|e|pg||
qx|o7|md|1SAQJ98764HD8643C5,SHKQT92DTCJT97642,ST2HA874DAQJ952CA|rh||ah|Board 7|sv|b|pg||
qx|o8|md|2SAKTHAKQT65DCQT64,S53H87DQJT965432C,SQ7HJ943D7CAK9852|rh||ah|Board 8|sv|0|pg||
qx|o9|md|3SAQ8H8DKCJT976432,SJ96HQ954DAJ7632C,ST72HAKT7632D984C|rh||ah|Board 9|sv|e|pg||
qx|o10|md|4SJHQ654DQT86CAJ95,SKT987643HD75CKT8,SAQHJ9DAKJ9432CQ4|rh||ah|Board 10|sv|b|pg||

Some:
qx|o1|md|3SAQT764HJDAKQ96C5,S98HAK987642DJ32C,SHQT3DT5CAKJT9863|rh||ah|Board 1|sv|0|pg||
qx|o2|md|4ST9865HDKQJT532C7,SQJ732HJ872DA864C,SAHT943DCQJ965432|rh||ah|Board 2|sv|n|pg||
qx|o3|md|1SAJ963H9873D2CJ94,S7HKT64DAKQT864C7,SKQT8542HAQDJ73C8|rh||ah|Board 3|sv|e|pg||
qx|o4|md|2SAJT876432H83DCAQ,SK9HAQ652DK8CJ952,SQ5HJT7DQJ9CT8743|rh||ah|Board 4|sv|b|pg||
qx|o5|md|3SAK96532HT6DAJ7C4,ST8HJ83DKQT98432C,SJ7HKQ97DCKJ86532|rh||ah|Board 5|sv|n|pg||
qx|o6|md|4SJ85H86DT95CQT974,S2HQJ754D4CA86532,S4HKTDKQJ87632CKJ|rh||ah|Board 6|sv|e|pg||
qx|o7|md|1SHKQJ9653D3CJT952,SAKQJ8765HA87D4C4,ST42HTDA852CAK863|rh||ah|Board 7|sv|b|pg||
qx|o8|md|2S2HDKQJ9864CKQT97,S875HAKQJT74DACJ2,SAKQJT963H62DT7C8|rh||ah|Board 8|sv|0|pg||
qx|o9|md|3SQHK2DKT6CKQJ6432,SKT932HQJ9743DC75,S854HT865DJ3CAT98|rh||ah|Board 9|sv|e|pg||
qx|o10|md|4SKH5DAQ983CKT7632,S62HAJ8762DJT75C4,SAQT987543H94D62C|rh||ah|Board 10|sv|b|pg||
qx|o11|md|1SJ98HQT965DA9CK96,SA3HKJ83DCAJT8543,SKQT6542H2DK85C72|rh||ah|Board 11|sv|0|pg||
qx|o12|md|2S974HAKQJT762D76C,SAK53H5DQ93CKQJ32,SH43DT842CAT98654|rh||ah|Board 12|sv|n|pg||

Some:
qx|o1|md|3SAKQ8653H432DCQT5,ST974H9DAQ753CA64,SHKDKT862CKJ98732|rh||ah|Board 1|sv|0|pg||
qx|o2|md|4S6HAQJT963DTCA974,SAK98732H4D9532C8,SQJ54HK872DCKQJT6|rh||ah|Board 2|sv|n|pg||
qx|o3|md|1SJHQ6DAT9876532CT,SAQT9843H92D4CQ73,SK752HT87543DQC86|rh||ah|Board 3|sv|e|pg||
qx|o4|md|2SK983HJ74DAKJ873C,SAJT76542HDQ64CK6,SHT953DT52CAQJT54|rh||ah|Board 4|sv|b|pg||
qx|o5|md|3ST982H983DQ4CAQ82,SA7HKDJ976532CK75,SQJ4HAQJT76542DC4|rh||ah|Board 5|sv|n|pg||
qx|o6|md|4SAQJ863H653D5C987,SHQJ9842D7CAJT432,S2HT7DAKQJT8643CQ|rh||ah|Board 6|sv|e|pg||
qx|o7|md|1SHK3DAJ642CKQJT94,SAKT97432HDQ7C532,SJ8HAQT87542DK8C8|rh||ah|Board 7|sv|b|pg||
qx|o8|md|2SK542HKT4D3CJ8432,SAJT86HDAQT4CKQ75,SQ973HAQ8763DCA96|rh||ah|Board 8|sv|0|pg||
qx|o9|md|3SQ976542HDAJ85CK9,STHKQT74DKQ942CA6,SHA932D6CQJ875432|rh||ah|Board 9|sv|e|pg||
qx|o10|md|4SK875HJ2DAKT975C4,SQJT943HDQ4CAJ752,SHAQT87DJCKQT9863|rh||ah|Board 10|sv|b|pg||
qx|o11|md|1ST43HQ76D765432C8,SQJ98652H843DAQ8C,SAH9DKT9CAJT97652|rh||ah|Board 11|sv|0|pg||
qx|o12|md|2SK75HT83DJCAK9654,SQ963HAK742DCQT87,SAJT842HJ5DT95C32|rh||ah|Board 12|sv|n|pg||
qx|o13|md|3S8HAKJT953D9853CQ,SQJH864DKQ764CKJ8,S42HQ72DJCAT97652|rh||ah|Board 13|sv|b|pg||
qx|o14|md|4SAT7HAKT98763D8CJ,SKQJ542HJ54DCK753,S96HDKQJT732CQT82|rh||ah|Board 14|sv|0|pg||
qx|o15|md|1SAKQ987532HJDKCJ6,S6HQ32DQT954CAK82,SJT4H94D876CQ9543|rh||ah|Board 15|sv|n|pg||
qx|o16|md|2SK862HAT96DQT2CJ6,S3HJD8CKQT9875432,SQ754HQ432DA543CA|rh||ah|Board 16|sv|e|pg||
qx|o17|md|3SJT754HT764DJCKT4,SKHAKJ98532DT4CJ8,SQHDKQ97632CA9653|rh||ah|Board 17|sv|0|pg||
qx|o18|md|4SKJ6HAQ9852DQ96C9,ST9732HK3DAJ532C4,S8H6D8CKQJT876532|rh||ah|Board 18|sv|n|pg||
qx|o19|md|1ST5H643DAQT8732C8,SAKJ976432HT5DJCA,S8HK982D64CT76532|rh||ah|Board 19|sv|e|pg||
qx|o20|md|2ST72HK4DQJ876432C,SKJ53HQD9CAKQJ862,SAQ94HT73DT5CT973|rh||ah|Board 20|sv|b|pg||

[pieh-ejdsch]

Hello sajjansinghania,

this may help You.

Code: Select all

@echo off
setlocal
set prompt=$g$s
 rem ich verschiebe Versionierte Backups in GOOD und gleiche in den Eimer
 rem  hier: Kontrolle auf doppelte Backup Dateien
:: --------------------------------------------------------------------------
 rem Folder 

pushD "D:\CMD - Scripte.Backup(Versions)"

:: --------------------------------------------------------------------------
set doubles=TRASH
set versions=GOOD
echo off
for %%i in (%doubles% %versions%) do if not exist "%%i\" ( md "%%i"
  if errorlevel 1 exit /b 1
)
for /f "delims==" %%i in ('2^>nul set _') do @ set "%%i="
set /a countFiles=countTrash=0

 rem es werden nur Dateien gleicher Größe miteinander Verglichen
 rem gleiche Dateien (errorlevel = 0) werden markiert/verschoben/entfernt
 rem System Anwendbar auf (Vergleich/Erstellung) Backup 

for %%i in (*) do ( setlocal enabledelayedexpansion
  for /f "usebackQtokens=1,2*" %%a in ('!countFiles! !countTrash! !_%%~zi!') do (
    endlocal
    set /a countfiles+=1
    set "EX="
    for %%d in ( %%c ) do if NOT defined EX (
      >nul fc "%%i" "%%~d" && (
        echo  weg  %%i
        >nul move "%%i" TRASH
        set /a countTrash+=1, countFiles-=1
        set /a EX=1
      ) 
    )
    if NOT defined EX set _%%~zi= %%c "%%i"
    title  Files  %%a  Trash  %%b
  )
)
title  Files %countFiles% trash %countTrash%  done
for /f "delims==" %%i in ('2^>nul set _') do @ set "%%i="
pause
exit /b

Phil

[sajjansinghania]

this works very well Sir but i wish to check files of different sizes also to find if same text appears. Please help. Tx

[Aacini]

In your first post you said: "My need is to check contents of each file and to identify 100% duplicate files". Now you changed the requirement to "i wish to check files of different sizes also to find if same text appears". This is an entirely different problem...

I strongly suggest you to read the first topic of this forum...

Antonio

[sajjansinghania]

@Antonio Sir,
as suggested by you i read viewtopic.php?f=3&t=6108. your right Sir, i posted wrong therefore i apologize. i am happy with whatever given by pieh-ejdsch Sir.
in any case, if files of different sizes can be checked without any major tweak would be welcome.
Thank you once again DosTips