TESTBLOCK(P) Julian H. Stacey, http://www.berklix.com TESTBLOCK(P) NNAAMMEE testblock - Test block input & output to test for media &/or driver errors. SSYYNNTTAAXX tteessttbblloocckk [-b block_size] [-d data_byte] [-f fail_size] [-i] [-I] [-l byte_limit] [-n] [-s seconds] [-r] [-v] [-w] [-0] -- or file[s] or special_file[s] DDEESSCCRRIIPPTTIIOONN Writes and/or reads test data blocks from &/or to a special file, normal file(s), or a pipe. Used to test IO on tape cassette, floppy drives, & hard drive partitions etc. Verifes media, dirty heads, faulty device drivers, etc, without the necessity of finding space on disc file systems to store large reference data images (such as tape sized comparison files). (A separate VSL/Berklix tool `valid' also exists, designed purely for floppy disc checking & recovery.) In the simplest case, data blocks are written till end of medium (or until file system is full **), then the written image is read & compared with a _t_e_s_t_b_l_o_c_k generated internal equivalent of what the data blocks should contain. ** If you are deliberately trying to fill your disc partition, eg to - over-write any previously used but now un-allocated blocks that might still have copies of secret information, Or - So the disk detects & patches out flakey sectors. (& so the OS pauses & hangs on IO now when it doesnt matter, rather than later when you may need disk performance), then Note: - If you are running on a BSD Unix such as FreeBSD you may need to either become root to fill the entire disc partition, or get your sys admin to first "umount /dev/BLA; tunefs -m 0 /dev/BLA; mount /BLA" if you are running on a nasty Microsoft file system such as FAT (& that is the norm for USB memory sticks), MS only allow up to 4Gig files so you may have to write numerous files to fill your disc. Write yourself a trivial batch script to do that. eg testblock -n -w junk000 testblock -n -w junk001 testblock -n -w junk002 etc If running on Microsoft, you should also remove all the files in a directory called [something like, name forgotten) .Recovery. Regardless if running on Unix, Microsoft or some other OS, after filling most of disc partition using _t_e_s_t_b_l_o_c_k with a large block size, eg by _t_e_s_t_b_l_o_c_k -w -b 1m junk_big_blocks For security to wipe the last bytes, you should then fill All remaining blocks by eg _t_e_s_t_b_l_o_c_k -w -b 1k junk_last_small_blocks Or better _t_e_s_t_b_l_o_c_k -w -b 512 junk_last_small_blocks DDAATTAA TTEESSTT PPAATTTTEERRNN Bytes in the first _t_e_s_t_b_l_o_c_k data block use an incrementing range of values, where each 256 bytes normaly start 0x00, 0x01, 0x02 etc, up to 0xFF, then the next 256 bytes repeat 0x00 0x01 etc, until the end of block. (The 0x00 start value can be altered with the -d option). The next _t_e_s_t_b_l_o_c_k data block starts 0x01, 0x02 etc. (The size of the _t_e_s_t_b_l_o_c_k block is controlled by the -b option). OOPPTTIIOONNSS All parameters may be in any order relative to each other. All parameters must precede all file names. Parameter default values are shown in brackets. ---- [[ooffff]] Pipe usage (off by default). See Output section. --vv [[ooffff]] Verbose. --rr [[oonn]] Read mode. On by default, but off if -w is invoked, unless -r is also specifically invoked. --ww [[oonn]] Write mode On by default, but off if -r is invoked, unless -w is also specifically invoked. --ii Ignore single errors instead of aborting file test immediately. Adjacent (ie double) errors will still cause a file test abort. --II Ignore bursts of read errors. Used to explore a USB ram stick to find another live section beyond a dead section. eg { my light brown wood effect stick labelled HSM: "vendor" "0x0000"; "product" "0x7777"; "devclass" "0x00"; "devsubclass" "0x00"; "release" "0x0101"; "intclass" "0x08"; "intsubclass" "0x06"; http://www.berklix.com/~jhs/src/bsd/fixes/FreeBSD/src/jhs/etc/devd/berklix.conf } --nn [[ooffff]] - Suppress the rewind that _t_e_s_t_b_l_o_c_k normally does both before write and/or read. - Essential for both virgin tapes, & tapes previously written with a different block size or density (where the drive can for instance write 150M or 525M density)). - If you use this option, you are relying on the tape supplier, &/or drive electronics having correctly rewound the tape after insertion. - This option is useful if you want to call _t_e_s_t_b_l_o_c_k from a script, to test the remainder of the tape, after initially skipping a tape label or other contents. - If -w & -r are both selected with -n, a caution is printed as the action required of _t_e_s_t_b_l_o_c_k seems strange. --bb BBlloocckk__SSiizzee [[6600kk oorr 88kk]] Change block size. ( Default block size if i386 is defined (as it is in some FreeBSD on older non AMD64 machines) or if amd64 is defined: 60K, else 8K (for Symmetric 375 (a BSD4.2 running Symmetrix http://www.berklix.com/symmetric/ ) etc) ). ((******)) Block Sizes are determined by the _t_e_s_t_b_l_o_c_k_._c program source, or by option -b. Block size is Not as per default system wide include/ header files. _t_e_s_t_b_l_o_c_k -v will tell you what size it will use by default. It assumes a block size that should be reasonably efficient for a hard disk file. HOWEVER if the file is name of device of a USB memory stick, the block size will be wrong, the program will run very slow, & it will wear the stick un-necessarily, so set a blocksize with eg "-b 64k" or "-b 256k". (The reason for this is that USB sticks allocate memory in big blocks, & if the program writes just a small block, the operating system has to read a Big block from stick, modify a small block within the big block, write out the big block, then mostly repeat, reading in the same Big block again, modifying the next small block along, writing out the big block again, etc etc.) Example: On FreeBSD 12.3-RELEASE, this program defaults to block size Block size 8192 (0x2000). Even though for both /dev/da2 (a USB memory stick) & da3 (a SATA hard disk via a USB adapter) _s_t_a_t_v /dev/da2_or_3 reports "Optimal I/O 4096". 2023-01-31 garyj@ with a 12 Terabyte SATA. on an external USB-3.1 enclosure, writing, saw 200 MiB/s, on motherboard main SATA connector about 230MiB/s with -b 1m. With -b 10m it was slower. --ll mmaaxx__lliimmiitt [[00 == nnoonnee]] Set a maximum byte count limit. ((******)) --ff ffaaiill__lliimmiitt [[00 == nnoonnee]] Set a fail byte count limit. User specified threshold (in bytes), below which, we dont bother to do a read after write (typically for a suspect/unknown tape being tested before being discarded as too short) (0=unset, ie never inhibit read after write). ((******)). --ss sseeccoonnddss [[00]] Number of seconds sleep between a write & then a read. (some tape drive need a sleep else will error). --dd ddaattaa__bbyyttee [[ooffff]] Every byte in every data block is forced to have the fixed hexadecimal value data_byte (rather than the normal incrementing range of values, where the first _t_e_s_t_b_l_o_c_k data block starts 0x00, 0x01, 0x02 etc). -d over-rides -0. --00 [[ooffff]] This option forces each block to start at 0. ( Normally the value of byte zero in each succesive block is incremented by 1 from the value of byte 0 in the previous block, thus block 0 byte 0 has offset 0, block 5 byte 0 has value 5, block 7 byte 3 has value 10. ) To check data created by _t_e_s_t_b_l_o_c_k versions prior to V2.0 96.04.02, you must use this flag. ((******)) Note: If the last character of the size is a 'g' or a 'G' (for Giga), size is multiplied by 1024 * 1024 * 1024; If 'm' or 'M' (for Mega), by 1024 * 1024; If 'k' or 'K' (for Kilo), by 1024; If 'b' or 'B' (for Blocks), by 512; If 'w' or 'W' (for Words), by 2. IINNPPUUTT aanndd OOUUTTPPUUTT If -r and -w are both specified, then writing is done before reading. You cannot simultaneously use all these three options -- & -w & -r. (as this would first write the output pipe, then read the input pipe, which would be rather weird, and not make much sense). Both and are used by _t_e_s_t_b_l_o_c_k_. If _t_e_s_t_b_l_o_c_k has set itself into `output to pipe mode' (by use of -- & -w), all diagnostic and error reports are sent to , otherwise only syntax/invocation errors are to , and media/file access errors are sent to (to ease error logging under a variety of different command interpreters). _T_e_s_t_b_l_o_c_k rewinds tapes before writing them by opening then closing the file, thus trying to ensure the tape is rewound before use, in case a tape has not been rewound after last use. EEXXAAMMPPLLEESS Example 1 _t_e_s_t_b_l_o_c_k -b 1k -l 800k Explanation 1 Write and then read a standard format Symmetric floppy, of capacity 800k bytes, using a block size of 1k bytes, testing the floppy media (and destroying any previously existing data of course). Example 2 Under FreeBSD-2.1 (CD Release): testblock -n -v -b 256w /dev/rsd2 Explanation 2 This will write then read /dev/rsd2, using a block size of 512. This will test the entire disc, first destroying the MBR / 4 Fdisk slices info, & then continuing to overwrite rest of drive sd2. _T_e_s_t_b_l_o_c_k report back the size of the entire physical disc, in the form of a `fail to write more than (example) 200 Meg. (but if you just want to know the size, on FreeBSD it's much quicker to use _d_m_e_s_g instead. LLIIMMIITTAATTIIOONNSS _T_e_s_t_b_l_o_c_k is not a standard Unix command, it is unique to JHS/VSL systems with the Berklix overlay. Msdos users wishing to validate tapes must supply their own device driver, making the drive available as a file name to _t_e_s_t_b_l_o_c_k_. _T_e_s_t_b_l_o_c_k does not check what sort of name it is given, so do not give it a `non rewind on completion' tape drive special file name, if you intend to do a write+read test. UUSSEE IINN SSYYSSTTEEMM DDEEBBUUGGIINNGG _T_e_s_t_b_l_o_c_k is a useful system debuging tool. _T_e_s_t_b_l_o_c_k was developed because tar & dd wouldn't work reliably with the author's `Symmetric' development host + TEAC cassette streamer. _T_e_s_t_b_l_o_c_k drove the streamer OK, thus proving the streamer was OK, & that the non streaming problem was associated with interaction with winchester drive activity. _T_e_s_t_b_l_o_c_k can be used to prove if your device/driver combination does not do an adequate read after write check (the author's `Symmetric' development host consistently wrote 40M, but could only read 20M byte, on some cassette tapes). _T_e_s_t_b_l_o_c_k was used by the author (04/96) to debug a problem involving FreeBSD sd, Adaptec 1542A, intermittent corruptions of 8 * 0xFF at beginning of 0x1000 boundaries. _T_e_s_t_b_l_o_c_k is still in use 2023 to test media ( eg 2023-01: garyj testing a 12 Terabyte SATA disk). _T_e_s_t_b_l_o_c_k can be run on all of raw devices, partitions, & files within file systems. SSEECCUURRIITTYY // FFAASSTT EERRAASSUURREE Both DOS & Unix, `del file' & `rm file' do not really delete file data, but merely deallocate blocks, (viz. Norton undelete now available on SCO Unix), so invaders/spies/crackers can still read your `erased' data blocks ... until you erase unused blocks. _t_e_s_t_b_l_o_c_k can erase large discs quickly, either used on raw physical devices, or within file systems, to erase all de-allocated spare free- chain blocks. Removable disk packs & USB disks are particular instances where _t_e_s_t_b_l_o_c_k could be used to pack erase the disc before re-issue to someone else. Making disc devices not readable by normal users is also a sensible initial precaution, but deleting data on the free chain is easy with a _/_u_s_r_/_l_i_b_/_c_r_o_n_t_a_b + _t_e_s_t_b_l_o_c_k combination. _t_e_s_t_b_l_o_c_k_, if used with the correct blocking factor, can be faster on some discs than a low level format. Some systems do not allow low level formatting except from the eprom/monitor, whereas _t_e_s_t_b_l_o_c_k can even be run remotely, if your machine has been captured, but not taken off the net (eg embassy invasion perhaps ?) A pre-prepared panic script could obliterate in parallel a prioritised list of discs rather than just the single disc at a time that system/eprom monitor + human operator interaction requires. ( If one is truly paranoid, even after testblock, I suspect that if discs that are physically captured, they can be read by special disk diagnostic machines, that read marginal edge-of-track data, by increasing read head op amp sensitivity, and realigning heads to read the reduced strength stray field down the edge of old tracks. A standard tecnique of CIA type people around r before early 1980s I recall ) PPRROOGGRRAAMM IINNTTEERRNNAALLSS _T_e_s_t_b_l_o_c_k checks its `C' read(2) and write(2) calls not merely looking for a negative return code error, but positively checking that the number of bytes read/written is exactly correct. This has been used to remind the author that although a Unix command such as dd if=/dev/rmt1 obs=64k | testblock -r -- -v -b 64k might be expected to work, it will not, because a Unix kernel typically only passes up to 4k bytes at a time down pipes, further if you dont know the maximum pipe block size on your system, issuing several different size variants of the above command will determine it. A test in Jan 2015 on a 64bit AMD64 FreeBSD-9.2-RELEASE with testblock -n -w -b 1m -l 10g junk showed it succesfully working beyond 32bit signed integer 2G limit, eg it generated a 10G (10737418240 byte) file. EEXXIITT CCOODDEE 0 if no errors occur, else number of errors detected before aborting. BBUUGGSS - Writes blocks, so _t_e_s_t_b_l_o_c_k _-_l _1_7_0 _-_b _1_0_0 writes 200 bytes, and causes _t_e_s_t_b_l_o_c_k to think an error has occured when it can not read the last 30 bytes. On FreeBSD current @ 96.03 , the rwt device stream writes at 16K, but on reading back does not stream, this is perhaps a fault of the wt driver software.) Should (but does not yet) default to the prefered block size of the device (st_blksize), referenced in _m_a_n _2 _s_t_a_t_, (as displayed by my _s_t_a_t_v_(_1_) command). _t_e_s_t_b_l_o_c_k _-_v _-_n _-_w _/_d_e_v_/_d_a_3 _-_b _6_4_k Will not report an error until after it has run for half a day, at an inefficient small block size, & completed /dev/da3 & then fails to access a file called ./-b, so ensure you invoke as eg: _t_e_s_t_b_l_o_c_k _-_v _-_n _-_w _-_b _1_m _/_d_e_v_/_d_a_3 I can't be bothered to add code to check all files exist before starting (& indeed that would break the Unix premise of least surprise as no other commands do test ahead), so classify this as a feature not a bug :-) This manual needs restructuring. HHOOSSTT OOPP SSYYSSTTEEMM This utility runs on Unix UCB BSD 4.2 + 50/60M Cassettes, Msdos 3.2 (files only), UNIX SVR4 i486 + Tandberg QIC525, all FreeBSD versions. testblock_32bit.exe compiled by gj Jan 2015 for Microsoft Windows 7 & 8. Should work under msdos on tape drives, if integrated with Adaptec ASPI for DOS SCSI Tape Drivers, but not tried. CCOOPPYYRRIIGGHHTT Program Copyright Julian H. Stacey, Munich, 14th March 1990. (Yes, a program that's been useful for 32+ years :-) Document Copyright Julian H. Stacey, Munich, 15th March 1990 - Jan 2015. See the file author.jhs. HHIISSTTOORRYY Program written to test suspect hardware & driver software, on my Symmetric 375 BSD4.2 system with bad scsi cassette tape using restore: partial block read: 10239 should be 10240 abort? [yn] n partial block read: 1 should be 10240 abort? [yn] FFIILLEESS /dev/rtape Default Unix file to read and write (administrator should create a symbolic link to device name such as rmt8). if tteessttbblloocckk writes to a named normal file, the ffiillee command (as tested on FreeBSD-12.3-RELEASE reports: "Targa image data - RGBA (1027-1541) 3340 x 3854 x 16 +2312 +2826 - 1-bit alpha - right" which is not true. SSEEEE AALLSSOO vvaalliidd((pp)) For tape capacities see: http://www.berklix.com/~jhs/standards/qic http://www.berklix.com/~jhs/txt/how-to-erase.html 4th Berkeley Distribution Manual V2.15 for Program V2.15 TESTBLOCK(P)