Combine Text Searches / Strings to a specified .cvs format

[booga73]

Code: Select all

for /f "tokens=7*" %a in ('findstr /c:"NIST SP 800-53 Revision 4 ::" test1.txt') do @echo %a %b
for /f "tokens=9*" %a in ('findstr /c:"Vuln ID:" test1.txt') do @echo %a

Hello DosTips,

I have a text file which I need 2 items pulled from. The 2 items I'm requesting to retrieve:

Item1 = Vuln ID: referenced number
Item2 = NIST SP 800-53 Revision 4 :: CM-6 b, simply the value which comes after "NIST SP 800-53 Revision 4 ::" which is not a value always the same for Item1.



Each "V" number is unique, the number never is the same. Item 2 appears some lines down from Item1. Item 2 sometimes appears multiple times for Item1.

For example, in the example text provided, Vuln ID: V-1070 has NIST SP 800-53 Revision 4 :: CM-6 b showing. But for Vuln ID: V-1089, there are multiple listing for "NIST SP 800-53 Revision 4 ::".

In retrospect, what I am requesting support is to combine my batchs attempt from above to read the sample text and output into the following file .cvs format:

This is the example of how the text output display i'm requesting to appear as.

Capture1.JPG (16.73 KiB) Viewed 7387 times

I'm running Windows10 if this helps too.

I can't presently upload or attached my test1.txt file (error: invalid file extension: test1.txt); however, I'll try to copy/paste a portion of my text file. If needed, I can be reached via private message if you need a complete copy of the text file..


thank you for your concerted effort, very respectfully, Booga73


contents of the test1.txt text file:

Code: Select all

Windows 7 Security Technical Implementation Guide  
Rule Title:  Physical security of the Automated Information System (AIS) does not meet requirements. 
STIG ID: 1.001  Rule ID: SV-24996r1_rule  Vuln ID: V-1070 
 
Discussion:  
Inadequate physical protection can undermine all other security precautions utilized to protect the system. 
 
Documentable: No 
 
Responsibility:  
System Administrator 
 
Check Content:  
Interview the SA to determine if equipment is located in an access controlled area. 
  
Note:  Servers will be located in rooms, or locked cabinets, that are accessible only to authorized systems personnel.  Authorized user access 
should be verified at two points (i.e. building access and server room).  
 
Fix Text:  
Relocate equipment to a controlled access area. 
 
CCI: CCI-000366 
The organization implements the security configuration settings. 
NIST SP 800-53 :: CM-6 b 
NIST SP 800-53A :: CM-6.1 (iv) 
NIST SP 800-53 Revision 4 :: CM-6 b 
 
 

Windows 7 Security Technical Implementation Guide  
Rule Title:  Shared user accounts must not be permitted on the system. 
STIG ID: 1.008  Rule ID: SV-25000r2_rule  Vuln ID: V-1072 
 
 
 
Discussion:  
Shared accounts (accounts where two or more people log on with the same user identification) do not provide adequate identification and 
authentication.  
 
Documentable: No 
 
Responsibility:  
System Administrator 
 
Check Content:  
Determine if any shared accounts exist.  If no shared accounts exist, this is NA. 
 
Any shared account must be documented with the ISSO.   
 
Note:  As an example, a shared account may be permitted for a help desk or a site security personnel machine, if that machine is standalone and 
has no access to the network. 
 
Fix Text:  
Create or update shared accounts documentation that minimally contains the name of the shared account(s), the system(s) on which the accounts 
exist, and the individuals who have access to the accounts. 
 
CCI: CCI-000764 
The information system uniquely identifies and authenticates organizational users (or processes acting on behalf of organizational users). 
NIST SP 800-53 :: IA-2 
NIST SP 800-53A :: IA-2.1 
NIST SP 800-53 Revision 4 :: IA-2 
 
 

Windows 7 Security Technical Implementation Guide  
Rule Title:  Permissions for event logs must conform to minimum requirements. 
STIG ID: 2.001  Rule ID: SV-25133r3_rule  Vuln ID: V-1077 
 
 
 
Discussion:  
Maintaining an audit trail of system activity logs can help identify configuration errors, troubleshoot service disruptions, and analyze compromises 
that have occurred, as well as detect attacks. 

Documentable: No 
 
Responsibility:  
System Administrator 
 
Check Content:  
Verify the permissions on the event logs.  Standard user accounts or groups must not have access.  The default permissions listed below satisfy 
this requirement. 
 
Navigate to the log file location.  The default location is the "%SystemRoot%\System32\winevt\Logs" directory. 
For each log file below, right click the file and select "Properties". 
Select the "Security" tab. 
Select the "Advanced" button, then "Continue", and respond to any UAC prompts. 
 
Log Files: 
Application.evtx 
Security.evtx 
System.evtx 
 
Permissions: 
Eventlog - Full Control 
SYSTEM - Full Control 
Administrators - Full Control 
 
If the permissions for the file are not as restrictive as those listed, this is a finding. 
 
If the organization has an "Auditors" group from previous requirements, the assignment of Full Control permissions to this group would not be a 
finding. 
 
Fix Text:  
Maintain the permissions on the event logs.  Standard user accounts or groups must not have access.  The default permissions listed below 
satisfy this requirement. 
 
Navigate to the log file location.  The default location is the "%SystemRoot%\System32\winevt\Logs" directory. 
For each log file below, right click the file and select "Properties". 
Select the "Security" tab. 
Select the "Advanced" button, then "Continue", and respond to any UAC prompts. 
 
Log Files: 
Application.evtx 
Security.evtx 
System.evtx 
 
Permissions: 
Eventlog - Full Control 
SYSTEM - Full Control 
Administrators - Full Control 
 
If the organization has an "Auditors" group from previous requirements, this group may be assigned Full Control. 
 
If the location of the logs has been changed, when adding Eventlog to the permissions, it must be entered as "NT Service\Eventlog". 
 
CCI: CCI-000162 
The information system protects audit information from unauthorized access. 
NIST SP 800-53 :: AU-9 
NIST SP 800-53A :: AU-9.1 
NIST SP 800-53 Revision 4 :: AU-9 
 
CCI: CCI-000163 
The information system protects audit information from unauthorized modification. 
NIST SP 800-53 :: AU-9 
NIST SP 800-53A :: AU-9.1 
NIST SP 800-53 Revision 4 :: AU-9 
 
CCI: CCI-000164 
The information system protects audit information from unauthorized deletion. 
NIST SP 800-53 :: AU-9 
NIST SP 800-53A :: AU-9.1 
NIST SP 800-53 Revision 4 :: AU-9 
 
 

Windows 7 Security Technical Implementation Guide  
Rule Title:  The required legal notice must be configured to display before console logon. 
STIG ID: 3.011  Rule ID: SV-25046r5_rule  Vuln ID: V-1089 
 
 
 
Discussion:  
Failure to display the logon banner prior to a logon attempt will negate legal proceedings resulting from unauthorized access to system resources. 
 
Documentable: No 
 
Responsibility:  
System Administrator 
 
Check Content:  
Analyze the system using the Security Configuration and Analysis snap-in. 
Expand the Security Configuration and Analysis tree view. 
Navigate to Local Policies &gt&gt Security Options. 
 

 
The policy referenced configures the following registry value: 
 
Registry Hive: HKEY_LOCAL_MACHINE 
Registry Path: \SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\ 
 
Value Name: LegalNoticeText 
 
Value Type: REG_SZ 
Value: See message text above 
 
Fix Text:  
Configure the policy value for Computer Configuration 

CCI: CCI-000048 
The information system displays an organization-defined system use notification message or banner before granting access to the system that 
provides privacy and security notices consistent with applicable federal laws, Executive Orders, directives, policies, regulations, standards, and 
guidance. 
NIST SP 800-53 :: AC-8 a 
NIST SP 800-53A :: AC-8.1 (ii) 
NIST SP 800-53 Revision 4 :: AC-8 a 
 
CCI: CCI-000050 
The information system retains the notification message or banner on the screen until users acknowledge the usage conditions and take explicit 
actions to log on to or further access. 
NIST SP 800-53 :: AC-8 b 
NIST SP 800-53A :: AC-8.1 (iii) 
NIST SP 800-53 Revision 4 :: AC-8 b 
 
CCI: CCI-001384 
The information system, for publicly accessible systems, displays system use information organization-defined conditions before granting further 
access. 
NIST SP 800-53 :: AC-8 c 
NIST SP 800-53A :: AC-8.2 (i) 
NIST SP 800-53 Revision 4 :: AC-8 c 1 
 
CCI: CCI-001385 
The information system, for publicly accessible systems, displays references, if any, to monitoring that are consistent with privacy accommodations 
for such systems that generally prohibit those activities. 
NIST SP 800-53 :: AC-8 c 
NIST SP 800-53A :: AC-8.2 (ii) 
NIST SP 800-53 Revision 4 :: AC-8 c 2 
 
CCI: CCI-001386 
The information system for publicly accessible systems displays references, if any, to recording that are consistent with privacy accommodations for 
such systems that generally prohibit those activities. 
NIST SP 800-53 :: AC-8 c 
NIST SP 800-53A :: AC-8.2 (ii) 
NIST SP 800-53 Revision 4 :: AC-8 c 2 
 
CCI: CCI-001387 
The information system for publicly accessible systems displays references, if any, to auditing that are consistent with privacy accommodations for 
such systems that generally prohibit those activities. 
NIST SP 800-53 :: AC-8 c 
NIST SP 800-53A :: AC-8.2 (ii) 
NIST SP 800-53 Revision 4 :: AC-8 c 2 
 
CCI: CCI-001388 
The information system, for publicly accessible systems, includes a description of the authorized uses of the system. 
NIST SP 800-53 :: AC-8 c 
NIST SP 800-53A :: AC-8.2 (iii) 
NIST SP 800-53 Revision 4 :: AC-8 c 3 
 
 

Windows 7 Security Technical Implementation Guide  
Rule Title:  The system must lockout accounts after 3 invalid logon attempts within a specified time period. 
STIG ID: 4.002  Rule ID: SV-25015r2_rule  Vuln ID: V-1097 
 
 
 
Discussion:  
The account lockout feature, when enabled, prevents brute-force password attacks on the system.  
 
Documentable: No 
 
Responsibility:  
System Administrator 
 
Third Party Tools:  
HK 
 
Check Content:  
Analyze the system using the Security Configuration and Analysis snap-in. 
Expand the Security Configuration and Analysis tree view. 
Navigate to Account Policies &gt&gt Account Lockout Policy. 
 
If the "Account lockout threshold" is "0" or more than "3" attempts, this is a finding. 
 
Fix Text:  
Configure the policy value for Computer Configuration &gt&gt Windows Settings  
 
CCI: CCI-000044 
The information system enforces the organization-defined limit of consecutive invalid logon attempts by a user during the organization-defined 
time period. 
NIST SP 800-53 :: AC-7 a 
NIST SP 800-53A :: AC-7.1 (ii) 
NIST SP 800-53 Revision 4 :: AC-7 a 
 
CCI: CCI-000205 
The information system enforces minimum password length. 
NIST SP 800-53 :: IA-5 (1) (a) 
NIST SP 800-53A :: IA-5 (1).1 (i) 
NIST SP 800-53 Revision 4 :: IA-5 (1) (a) 
 
CCI: CCI-002142 
The information system terminates shared/group account credentials when members leave the group. 
NIST SP 800-53 Revision 4 :: AC-2 (10)

[elzooilogico]

Code: Select all

for /f "tokens=7*" %a in ('findstr /c:"NIST SP 800-53 Revision 4 ::" test1.txt') do @echo %a %b
for /f "tokens=9*" %a in ('findstr /c:"Vuln ID:" test1.txt') do @echo %a

Hello DosTips,
...

let findstr find the data.

any time an id is found is stored in a var named ID[index], where index, will be 1,2,3...

until a new id is found, any other matching is then item 2 so it is appended to a var named str.

when a new id is found, str is copied to a corresponding str[index] var, then index is incremented, id[index] stores the new id, and str is erased

Now we only have to write results to a file. so the following is enough

Code: Select all

rem list view
>"%simpleOut%" (
  for /L %%1 in (1,1,!cnt!) do echo !ID[%%1]! !str[%%1]:~0,-1!
)

But write data as a table is more tricky, as each item 1 may contain multiple item 2

This time data has to be parsed data to fill the table cells. I think is (more or less) self explanatory if you know about batch programming

give this a try,

Code: Select all

@echo off
SetLocal EnableDelayedExpansion EnableExtensions

rem put desired names here
set "inputFile=test1.txt"
set "simpleOut=simple.txt"
set "tabularOut=tabular.txt"

rem adjust padding and columns to your needs
set /a padding=15, columns=8, hyphens=padding*columns
for /L %%1 in (1, 1, %padding%) do set "spaces=!spaces! "         & rem for padding columns
for /L %%1 in (1, 1, %hyphens%) do set "hyphenLine=!hyphenLine!-" & rem compose a separator

rem grab DEL char and build line eraser. This is only used to be user friendly
for /F "tokens=1,2 delims=#" %%# in ('"prompt #$H# & for %%@ in (1) do rem"') do set "DEL=%%#" & rem DEL
for /L %%a in (0,1,34) do set "delLine=!delLine!%DEL%%DEL%"

rem get data into variables, findstr will find data for us
set "str="
set /a cnt=0, max=0
for /F "usebackq tokens=1-5 delims=:" %%1 in (`findstr /ni /c:"Vuln ID" /c:"NIST SP 800-53 Revision 4" "%inputfile%"`) do (
  if not .%%5.==.. (
    rem must be item 1. get data and copy str into corresponding str[index]
    if not "!str!" == "" set "str[!cnt!]=!str!"
    set /a cnt +=1
    set "ID[!cnt!]=%%5"
    set "str="
  ) else if not .%%3.==.. (
    rem must be item 2
    set "str=!str!%%3,"
  )

)
rem get last data string.
if not "!str!" == "" set "str[!cnt!]=!str!"

rem list view
>"%simpleOut%" (
  for /L %%1 in (1,1,!cnt!) do echo !ID[%%1]! !str[%%1]:~0,-1!
)



rem table view is more tricky
rem compute loops / so we parse data by column blocks
set /a start=-(columns-1), stop=0, loops=cnt/columns, lastLoop=cnt%%columns
if %lastLoop% neq 0 set loops+=1


rem now build the table
>"%tabularOut%" (
  for /L %%$ in (0,1,%loops%) do (
    rem next line to be user friendly for large files
    <NUL set/p=!delLine!Parsing !stop! of !cnt!>con

    rem set start and end of column block
    set /a start+=%columns%, stop+=%columns%, line=0, max=0
    if !stop! gtr !cnt! set /a stop=cnt

    rem get maximun number of rows for this column block
    for /L %%a in (!start!,1,!stop!) do (call :split "!str[%%a]:~0,-1!", max)

    rem compose ID row
    for /L %%a in (!start!,1,!stop!) do (
      set "data=!ID[%%a]!%spaces%"
      set "header=!header!!data:~0,%padding%!"
    )

    rem parse item 2 list to build cells
    for /L %%# in (0,1,!max!) do (
      for /L %%a in (!start!,1,!stop!) do (
        set "data="
        for /F "tokens=1* delims=," %%x in ("!str[%%a]!") do (
          set "data=%%x"
          set "str[%%a]=%%y" & rem remove cell from list
        )
        set "data=!data!%spaces%"
        set "line%%#=!line%%#!!data:~0,%padding%!"
      )
    )
    rem now echo results and empty rows
    echo !header! & set "header="
    echo(%hyphenLine%
    for /L %%# in (0,1,!max!) do echo !line%%#! & set "line%%#="
    echo(&echo(&echo(
  )
)
echo(!delLine!Done. Found %cnt% items.& echo(
echo List view saved as  "%simpleOut%"
echo Table view saved as "%tabularOut%"& echo(
rem if running from explorer, wait some seconds
(((echo.%cmdcmdline%)|find /I "%~0")>NUL) && (timeout /T 7&echo/)
Endlocal
exit/b

rem split string into substrings based on delimiter
rem http://www.dostips.com/forum/viewtopic.php?f=3&t=6429#p41035
:split
SetLocal EnableDelayedExpansion
  set "line=%~1"
  set/a i=0
  set "x!i!=%line:,=" & set /A i+=1 & set "x!i!=%"
EndLocal & (if %i% gtr !%2! set /a"%2=%i%")
exit/B

[booga73]

Elzooilogico,

That is very amazing; thank you for your support. Both outputs are good.

very respectfully,
Booga73

[elzooilogico]

Elzooilogico,

That is very amazing; thank you for your support. Both outputs are good.

very respectfully,
Booga73

lucky you, this is something I faced before, so answer was pretty forward.

glad to see it helps, but next time I'd prefer you to show your efforts.

[Aacini]

Another approach:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set /A pad=15, col=0, i=0, lastI=0
set "spaces="
for /L %%i in (1,1,%pad%) do set "spaces= !spaces!"

rem Process the data found by findstr command
for /F "tokens=1-4 delims=:" %%a in ('findstr /C:"Vuln ID" /C:"NIST SP 800-53 Revision 4" test1.txt') do (
   if "%%d" neq "" (
      rem "Vuln ID" data: add a new column in line 1 always
      set /A colM1=col, col+=1
      set "data=%%d%spaces%"
      set "line[1]=!line[1]!!data:~1,%pad%!"
      rem Fill previous lines with same number of columns, up to the last line
      set /A i+=1
      for /L %%i in (!i!,1,!lastI!) do set "line[%%i]=!line[%%i]!%spaces%"
      rem Reset lines for new column
      set i=1
   ) else (
      rem "NIST" data: add as many lines as new NIST data to current column
      set "data=%%b%spaces%"
      set /A i+=1
      for /F %%i in ("!i!") do (
         rem If this line have not previous column: insert blank columns before
         if not defined line[%%i] for /L %%j in (1,1,!colM1!) do set "line[%%i]=!line[%%i]!%spaces%"
         set "line[%%i]=!line[%%i]!!data:~1,%pad%!"
      )
      if !i! gtr !lastI! set "lastI=!i!"
   )
)

rem Output accumulated results
for /L %%i in (1,1,%lastI%) do echo !line[%%i]!

Output:

Code: Select all

V-1070         V-1072         V-1077         V-1089         V-1097
CM-6 b         IA-2           AU-9           AC-8 a         AC-7 a
                              AU-9           AC-8 b         IA-5 (1) (a)
                              AU-9           AC-8 c 1       AC-2 (10)
                                             AC-8 c 2
                                             AC-8 c 2
                                             AC-8 c 2
                                             AC-8 c 3

Antonio

[elzooilogico]

@Antonio, I began with a script similar to yours, copying the contents of the test1.txt text file. Then I downloaded the test1.zip to test it against the real data. This file contains 146 IDs to parse, and (at least on my computer) notepad can't show 146 columns, it wraps lines, so result was a mess.

Despite the above, your code is amazinly fast!

[Aacini]

@elzooilogico,

I wasn't aware of that, and the OP did not indicated what to do when there are many columns. However, the modifications required in my code to generate an output with a limit in the number of columns are pretty simple:

Code: Select all

@echo off
setlocal EnableDelayedExpansion

set /A pad=15, cols=8,  dash=pad*cols, col=0, i=0, lastI=0
set "spaces=" & for /L %%i in (1,1,%pad%) do set "spaces= !spaces!"
set "dash=" & for /L %%i in (1,1,%dash%) do set "dash=-!dash!"

rem Process the data found by findstr command
for /F "tokens=1-4 delims=:" %%a in ('findstr /C:"Vuln ID" /C:"NIST SP 800-53 Revision 4" test1.txt') do (
   if "%%d" neq "" (
      rem "Vuln ID" data: add a new column in line 1 always
      set /A colM1=col, col+=1

      rem If number of columns reach the limit
      if !colM1! equ %cols% (
         rem Output accumulated results so far
         for /L %%i in (1,1,!lastI!) do echo !line[%%i]!& set "line[%%i]="
         echo/
         echo %dash%
         echo/
         set /A colM1=0, col=1, i=1, lastI=0
      ) else (
         rem Fill previous lines with same number of columns, up to the last line
         set /A i+=1
         for /L %%i in (!i!,1,!lastI!) do set "line[%%i]=!line[%%i]!%spaces%"
         set i=1
      )
      set "data=%%d%spaces%"
      set "line[1]=!line[1]!!data:~1,%pad%!"

   ) else (

      rem "NIST" data: add as many lines as new NIST data to current column
      set "data=%%b%spaces%"
      set /A i+=1
      for /F %%i in ("!i!") do (
         rem If this line have not previous column: insert blank columns before
         if not defined line[%%i] for /L %%j in (1,1,!colM1!) do set "line[%%i]=!line[%%i]!%spaces%"
         set "line[%%i]=!line[%%i]!!data:~1,%pad%!"
      )
      if !i! gtr !lastI! set "lastI=!i!"
   )
)

rem Output last accumulated results, if any
if defined line[1] for /L %%i in (1,1,%lastI%) do echo !line[%%i]!

Antonio