Table 1: Fraction of corpus files found in the NSRL, grouped by extension.

Table 2: Fraction of corpus files found in the NSRL, grouped by top-level directory name.

Table 3: Fraction of corpus files found in the NSRL, grouped by immediate directory name.

 

An issue with finding hash-code matches are the currently 225,430 records for zero-size files in the NSRL, all of which have the same hash value.  They must be stored separately and require exact matches on the file name to match to corpus files, for otherwise they slow processing greatly.

The operating system version for each drive in the corpus was inferred by collecting the best-matching NSRL entries for each file on the drive using just their file name and extension.  Assuming a file had K such matches with the best matching value, a weight of 1/K is added to the totals for each operating system version mentioned in those K matches.  The operating system version with the highest total over all files is assumed.  This approach reduces the chances of misidentifying a drive with legacy data from more than one operating system.

The NIST product codes (reference numbers of the software packages in which the files were originally found) were also tracked.  The product code from the best match in the NSRL data for each file is used, assuming that just a single file for a product is sufficient to indicate use of the product.  Drives averaged about 50 distinct product codes, not a large number.

5.2 Types of discrepancies between file names in NSRL and our corpus

NSRL-supplied file names were compared to their best matches in the corpus with the same hash value, and a program proposed possible explanations for discrepancies.  Hash collisions are an unlikely explanation since the probability of even a single occurrence of one on our corpus when using SHA-1 is .  But other explanations are possible.  Table 3 shows the counts of the best explanations calculated by our tool.  Some further explanation of the categories:

·  Some of the corpus file names are default names like "..".

·  Some NSRL file names appear to be temporary or preliminary, such as "__0X01FC" for "00000025.query" in the corpus and "_ECE3E78EB29F498DAD23059C0ED928E" for "erv43260.dll" in the corpus.  Some NSRL names appear to have appended hash values, like "gray.pf.2b708bc3_5b4d_47c0_bcc5_3e1bd2c51e5b" for "gray.pf" in the corpus.  These names should be updated in NSRL.

·  The NSRL name may have an extra file extension on the name used in the corpus, as for instance, "module1.xba.old" in NSRL and "module1.xba" in the corpus.  This suggests that NSRL obtained an early or restricted form of software.  The extra extensions should be deleted in NSRL.

·  The corpus may have an extra file extension on the name in NSRL, for instance "canon-s600.xml.nodrv" in the corpus matching "canon-s600.xml" in NSRL.  This suggests a different product uses a different name for the same software, a name that should be added to NSRL.

·  The file extension alone may differ on an update, as "netapi.os2" in NSRL which matches "netapi.dll" in the corpus.  This suggests additions to NSRL.

·  One file name may have added numbers to designate a version number, such as "setup[2].dll" in the corpus for "setup.dll" in NSRL.

·  One file name may have substituted underscores or exclamation points for the first or last character in the file name, e.g. "MDMGL004.IN_" in NSRL for "mdmgl004.inf" in the corpus, and "wz.pi!" in NSRL for "wz.pif" in the corpus.  These appear to be associated with file deletions as discussed in section 3.  The many cases of these suggest that a significant number of the NSRL files collected for hash values are in fact deleted files.  Computing hash values on deleted files could be risky for NIST as more than just the name of the file can be unreliable after deletion.

·  The file name in NSRL may be misspelled.  These were identified conservatively as cases where the NSRL file name was the same as the corpus file name except for a missing character, an added character, or a transposition of two adjacent characters.  Examples are "dirtbike.bmp" in NSRL for "dirt bike.bmp" in the corpus, and "gnome-dice2.png" in NSRL for "gnome-die2.png" in the corpus.  These appear to be errors in the NSRL.

·  The file name in NSRL may be mispunctuated, as for instance "which_2.sh" instead of "which-2.sh" in the corpus.  These appear to be errors in the NSRL.

·  One file name may be an abbreviation of the other, as for example "EVTMGMT.MDZ" in NSRL matching "event management.mdz" in the corpus.  These appear to be cases where NSRL had a preliminary version of software.

·  The name in NSRL may be a singular when the name in the corpus is a plural, or vice versa.  These appear to be errors in the NSRL.

·  The name in NSRL may be a translation of a name in a foreign language, as "conejo.wmf" on a Spanish-language corpus drive matching "rabbit.wmf" in NSRL.  These can be ignored.  This is discussed more in the next section.

·  Some hash values are for files serving multiple purposes under different file names.  Some are default files and some are common graphics.  See section 7.1.

·  Small files such as zero-size files may be accidentally identical to one another.

·  The file may have been copied and renamed.  This may be the case with some of the more radically different file names.  When done with commercial software it may suggest theft; when done with sensitive information it may suggest espionage.

·  The file name in the corpus may be in error.  Evidence for this was more frequent on deleted files where partial garbage collection could have destroyed the true file name.  See the discussion in section 6.

·  Someone may have manufactured a malware file to match a specific NSRL signature in the hopes of deceiving investigators into thinking it was an acceptable known file.  This possibility has been discussed in the literature, but the space of SHA-1 hash values is sufficiently large that engineering such a file with a specific functionality would be very difficult.  No evidence of this was found in our corpus.

 

Table 4 shows the counts on these discrepancies within the 11.8 million files of our corpus that matched hash values in NSRL.  The categories are designed to be mutually exclusive, so the more general categories only apply to items to which the more specific categories do not.

5.3 Foreign-language translation

Since our corpus is international, some discrepancies in file names are due to differences in languages.  This did not occur as often as might be expected, as much software uses English file names even when it does not originate in an English-speaking country.  Nonetheless, there were a significant number of translations in file names.

To check this, the Systran automated translation system was used.  It covers most of the foreign languages in our corpus, most importantly the Chinese, Spanish, German, and Arabic words.  The language is inferred as the predominant language of the country the drive comes from (only the country of origin is known).  File paths were split into component words, sent to Systran for a translation, and results reassembled into a path.  If Systran failed or returned the input, it was ignored.  But there were 1873 file paths for which it found something, mostly Spanish to English.  Systran is less successful on finding the proper syntax for translations, and permutations and rephrasings were not checked, so there were likely more acceptable matches than we counted.

6. Precision of the NSRL

An important question is the precision of the NSRL data, or the degree to which the data offered is correct.  Hash values are unlikely to be erroneous, but file name, size, and product type could be.

 

 

 

Table 4: Counts of file-name discrepancies between the NSRL and our corpus.

 

To test errors in file names, all hash values were collected which occurred at least five times in the corpus but whose NSRL name never occurred in the corpus.  Such cases are likely to be errors in file names because the space of SHA-1 hash values is so sparse that an incorrect one is unlikely to map to any files at all in the corpus.  There were 43,384 such hash values for our corpus.  There were simple explanations of some when the NSRL name was compared with the most popular name in the corpus.  The NSRL name was the same except for a final underscore in 15,865 cases, the NSRL name was the same except for a final exclamation point in 3,163 cases, the corpus name was embedded in the NSRL name in 903 cases, the NSRL name had additional characters in 1,535 cases, and there were 111 additional minor differences.  There were 4,363 cases of the same extension with very different file names, which were probably alternative names for the file because the odds of an incorrect name matching in extension are small.  There were 15,500 remaining cases where both the extension and filename did not match.  These should be investigated as possible errors in NSRL file names, or at least as suggesting a new additional file name in NSRL for that hash value.

Another question is whether the file sizes given in NSRL are correct.  So NSRL-reported file sizes were compared to those found for the same hash value in the corpus.  They differed on 7,461 corpus files, and usually by large amounts.  This was a surprise was because the size of a file should be easy to compute in building the NSRL, and the chances of a file of different length mapping to the same SHA-1 hash value are vanishingly small.  An example of a size discrepancy is the file test2.zeros of size 4096 in NSRL which had the same hash value as file AA00389B.71 of size 2490544 in the corpus.  The possible explanations:

·  This is a hash collision, unlikely given the size of the SHA-1 hash space.

·  The hash values could be incorrect.  This is unlikely since they were computed afresh for each file in both the corpus and the NSRL.

·  Nearly all the NSRL file sizes in these cases were powers of 2, so NSRL may be measuring a block size containing the file rather than the file itself.  File sizes that were powers of 2 occurred in only 2.2% of the corpus files with NSRL hash values, so this is cannot be a coincidence.

·  The file sizes that Fiwalk retrieved for our corpus files could be incorrect.  All the cases involved unallocated files in the corpus, which tend to have less reliable metadata. This explanation appears to be most likely.

Also investigated was whether there were possible errors in hash values given in NSRL by finding records in the corpus for hash values that never occurred in NSRL but which had the same name as a file in an entry in NSRL.  No such cases occurred.  This was surprising because file names can be short and there was a definite probability of a false alarm at least.  So there was no evidence of errors in NSRL hash values by this test.

Also checked was the product-code information provided by the NSRL by comparing our own group classification of the files in the corpus (based on lowest recognizable directory in the path of the file) with a classification based on the NSRL product code.  The latter was obtained by manually mapping the 887 distinct product descriptions in the NSRL to our groups.  Only 12.3% of the group classifications agreed, from which one concludes that the data is too different to be directly comparable.  For instance, Program Files/Microsoft Office/media/CntCD1/Photo1/ j0180794.jpg is classified as an image file by our taxonomy but as applications software by NSRL.  Both classifications are justifiable but not comparable.

NSRL also maps from product to product type.  One particularly interesting NSRL product type is "Hacker Tool" which covers malware.  To test its use, ClamAV antivirus software was run on a subset of 48 drives of our corpus, finding 6875 files containing signatures of malware.  Looking these up in NSRL found 93 items (35 distinct) with NSRL hash values.  In none of these cases did NSRL classify the file as a "Hacker Tool", and most were identified as "MSDN Library".  So NSRL completely failed at recognizing malware on these drives.  NSRL does not claim to be good at doing so, but users should not be misled by the presence of the "Hacker Tool" category in the NSRL classification.  More importantly, the presence of what appear to be 93 virus-infected files with hash values in the NSRL RDS is disturbing.  These could represent mistakes in identifying the virus file since the virus checker reported only the name of the file, not its path, and there were multiple files with the same name in these cases.  This is a subject for future study.

7. Recall of the NSRL

An even more important question is the recall (coverage) of the NSRL, or the fraction of appropriate files for which it has hash values.  This is somewhat subjective, since one can argue whether files only created after software is installed should be included, and if so, which.  Updates, configuration files, and default documents would seem important to include, but not temporary files.  NIST does not consider dynamically created files as part of its mandate, just files contained in the packaging of software.  But a 32.7% success rate in matching files in our corpus sounds low considering how few files a typical user creates themselves.

7.1 Multidrive files missed by the NSRL

The 67.3% of the corpus whose hash values were not recognized by NSRL include a variety of files.  Hash values that occurred on at least five different drives in the corpus provide relatively uncontroversial proposed additions to the NSRL hash values.  They provided 166,113 distinct hash values.  Some examples:

·  The file "machine.inf" of size 103496 occurs 40 times in the corpus with the same hash value, under both "WINNT/inf" and "WINNT/ ServicePackFiles/i386".  Judging by the path and extension, this appears to be part of the Windows NT operating system missed by the NSRL.

·  As mentioned in section 5, the same file can serve multiple purposes, and this often happens with basic graphics files.  For instance, the following names were used in the corpus for the same 56-byte GIF image: "BTN-DO~1.GIF", "TB_SRH~1.GIF", "DF_REV~1.GIF", "ICON_A~1.GIF", "RND03_~2.GIF", "vlog.idx", and "ESQUIN~1.GIF".

·  Another hash value of size 56 occurred 912,013 times in the corpus, usually under the name "." or ".." but in many different directories.  This appears to be default contents for a basic operating-system file.

·  Cache files can be identical when the system state did not change between writes.  If their contents originated from the operating system or the software, they may appear on multiple drives.  An example is a hash value of size 10 that appeared under many names in our corpus including "A0021284.ini", "A0021290.ini", "A0022464.ini", "A0022478.ini", "A0022490.ini", etc., and usually under "System Volume Information" in Windows.

A side benefit to finding multidrive hash values in the corpus is that they can indicate metadata errors in our drive imaging process.  For instance, "WINNT/inf/machine.inf" also occurred once listed as "WINNT/Fonts/Impact.ttf", a graphics file.  The latter is unlikely to be correct since a graphics file cannot serve as an INF executable; probably the drive imaging extracted the wrong metadata for a deleted file.  Such errors appeared on around 1% of the files with hash values occurring on five or more drives.  However, such errors do not affect the validity for exclusion of the hash values themselves that have been found many times since they occur so rarely.

7.2 Similarly named files missed by the NSRL

Other candidates for possible addition of hash values to the NSRL are files that are closely related to files already in the NSRL.  These may represent software versions that NSRL has not yet had an opportunity to analyze.  To test this, all cases in our corpus of files with hash values not known to NSRL were extracted where a match or a variant on that name with a NSRL hash value appears in the same directory (though not necessarily on the same drive).  This includes:

·  Identical file names and paths, like file "OLETHK32.DLL" found in directory "WINDOWS/SYSTEM" on different systems with different hash values.  These appear to be upgrades or patches and should be uncontroversial additions to the NSRL.  However, they could represent fake software or modifications of software by malware, so antivirus software should be run on them.  Such cases contributed 166,113 hash values from our corpus.

·  Names identical except for extension, like "eng.xtr" and "eng.lex" found in directory "Program Files/Hewlett-Packard/HP PrecisionScan/ISTech/OCR".  These are very likely related; though they could be data specific to a user, most software would use a different naming scheme.  These contributed 279,849 hash values from our corpus.

·  Names identical except for numbers just before the extension, like "taunto11.wav" and "taunto13.wav" in the directory "Games/Age of Empires/sound".  These appear to be parts of the same software package; perhaps the one without a hash value was added to the software after NSRL obtained a copy.  Although caching often numbers files, if one file has a hash value in the NSRL, the other is unlikely to be a cache.  These contributed 276,383 hash values from our corpus.

Altogether 937,570 additional hash values were found using both multidrive occurrences and similarities in names, enabling reduction of the corpus files for analysis by 2,559,749.

7.3 File-type filtering of files

The busy investigator may also want to exclude other kinds of files depending on the type of investigation.  Our classification of files based on extension and directory permits this to be done automatically.   Good candidates for such additional hash values are:

·  Files with executable extensions like "exe" and "dll", which were 16.1% of our corpus.  An exception would be investigations involving malware.

·  Files in operating-system directories such as "Windows" (30.1% of the corpus) or applications software directories such as "Program Files" (16.0% of the corpus).  But this loses user-related information stored their such as preferences and activity history which could be important in an investigation.

·  Configuration files (3.9% of the corpus by extension), since many of these are uninteresting defaults for a user.

7.4 Vendor coverage of the NSRL

Particularly useful for NIST to know are the applications-software directories in our corpus with large numbers of files not in NSRL, since it suggests vendors and products that they should target to improve the completeness of their collection.  To find this information, a tool was written to find all files in directories under "Program Files" in Windows, its variants and translations in languages of the corpus, and "bin" on Macintosh and Linux machines. It also sought vendor and product directories at the top level of the file system like "python26" since some vendors put software there too, particularly on older machines; but this only considered directories recognized in the group mapping described in section 3 as indicators of software, hardware, operating-system, and game directories, of which there are currently 402.

The metric used was the fraction of files in NSRL for all files under those directories and their subdirectories, categorized by the inferred vendor/product name (the first directory under "Program Files" or its variants, or the top-level directory name if that was used).  Table 5 shows a sample data for the corpus.  Vendors with low fractions are prime candidates for attention by NIST.  51.1% of the software directories (1718 out of 3360) in fact did not have a single file with a hash value in NSRL, and 74.3% of them had hash values for less than 10% of their files, so the NSRL clearly has some gaps.  Not all of these suggest action by NIST, as some of these files are old and unlikely to be encountered again, and some were data files.  But the percentages are worrisome.

Table 5: Sample product/vendor fractions of files having NSRL hash values.

 

8. Conclusions

Our analysis should give a clearer picture of the strengths and weaknesses of the NSRL RDS.  Its precision is very good, but its recall is imperfect and users should be aware of that.  Coverage was observed to be spread over a wide range of categories, and was not just confined to executables.  There were a surprising number of gaps in the coverage of NSRL in regard to vendors, and 74% of vendors in our corpus were substantially uncovered.  Coverage of malware was poor despite some misleading wording in the NSRL documentation.

Some errors and possibilities for improvement on auxiliary information of the NSRL were also found such as incorrect file names and unhelpful product descriptions.  Comparison of our corpus with NSRL did pinpoint some errors in our own drive imaging, most obviously on file sizes.

NIST considers that its mandate is to compute hashcodes from software downloads and media without running the software, but this is unreasonable in today's world of complex software installation procedures.  This paper showed that two simple kinds of additions to the NSRL RDS could improve its coverage by almost a million hash values without any added work of acquiring, mounting, and analyzing files.  While this is small fraction of the size of the RDS, few of these hashes could be obtained by NIST's traditional methodology of hiring interns to mount each software package and check it.  Commercial hash sets like that of bit9.com may be a better source of hash sets for users who can afford them.  However, our additional hashes are more consistent with NIST's philosophy for NSRL of finding files that are likely to occur frequently on drives.

This analysis has resulted in much useful data that will be provided to the NSRL managers, and since the analysis is automated, it can be run periodically to track the NSRL RDS.  The analysis software is general and will apply to any hash set that can supply associated file names and sizes.  Both the software, part of the Dirim suite of tools, and analysis results are freely available.

Acknowledgements

The views expressed are those of the author and do not necessarily represent those of the U.S. Government.  Thanks to Albert Wong and Riqui Schwamm.

References

Agrawal N, Bolosky W, Douceur J, and Lorch J.  A five-year study of file-system metadata.  ACM Transactions on Storage 2007; 3, 3: 9.

Buchholz F and Spafford E.  On the role of file system metadata in digital forensics.  Digital Investigation 2004; 1: 298-309.

Fisher K, Foster N, Walker D, and Shu K.  Forest.  A language and toolkit for programming with filestores.  Proc. ICFP, Tokyo, Japan 2011; 292-306.

Garfinkel S, Farrell P, Roussev, V, and Dinolt G.  Bringing science to digital forensics with standardized forensic corpora.  Digital Investigation 2009; 6: S2-S11.

Hoelz B, Ralha C, and Geeverghese R.  Artificial intelligence applied to computer forensics.  Proc. Security, Access, and Privacy, Honolulu, Hawaii 2009; 883-8.

Kim K, Park S, Chang T, Lee C, and Baek S.  Lessons learned from the construction of a Korean software Reference Data Set for digital forensics.  Digital Investigation 2009; 6: S108-S113.

Naiqi L, Zhongshan W, and Yujie H.  QuiKe: Computer forensics research and implementation based on NTFS file system.  Proc. Intl. Colloquium on Computing, Communication, Control, and Management, Guangzhou, China 2008; 519-523.

Richard G and Roussev V.  Next-generation digital forensics.  Communications of the ACM 2006; 49, 2: 76-80.

Rowe N and Garfinkel S.  Finding anomalous and suspicious files from directory metadata on a large corpus.  3rd International ICST Conference on Digital Forensics and Cyber Crime, Dublin, Ireland, 2011.

White D and Ogata M. Identification of known files on computer systems.  Retrieved February 17, 2012 from www.nsrl.nist.gov/Documents/aafs2005/ aafs2005.pdf (2005).