How To Quickly Generate A Large File On The Command Line (With Linux)There comes a time in every developers life where they need a data file for testing purposes and there are none handy. Rather than searching around for a file that fits your needs, the easiest thing to do is simply to generate one.
There are a number of reasons why you might want to generate a data file. For example, recently we needed to test the file upload functionality of a little application we were writing at work, using a whole range of files of different sizes (from < 1. Mb up to > 1. 00. Mb). Rather than hunt around for files that would fit the bill, it was a lot easier to just generate some.
Another reason might be when you need to test some functionality (e. Since you normally don't have files that are 1. Gb or more in size just lying around, generating some is probably a good way to go. Fortunately the Linux command line has all the tools we need to quickly and easily generate any kind of data file that we require (I am of course assuming that as a self- respecting developer you're using or at least have access to a Linux system : )).
![Create Or Append To File Ruby Create Or Append To File Ruby](https://ipowerthailand.files.wordpress.com/2010/06/kerio-setting-2.gif)
- An even quicker way to generate a really big file that contains only null characters is by using the seek parameter, thus generating a sparse file. Here we create an one terabyte file in a fraction of a split second: dd if=/dev/zero of=one-tera-byte-file bs=1 count=0.
- In my previous posts, we discovered How to Use HTML5 File Drag & Drop, Open Files Using JavaScript and Asynchronously Upload Files Using Ajax. In the final part of this series, we cover the most exciting part of the process: graphical progress bars! File upload progress bars provide essential user.
- Ruby File I/O, Directories - Learning Ruby in simple and easy steps - A beginner's tutorial containing complete knowledge of Ruby Syntax Syntax, Classes,, Variables, Comments, Loops, File, I/O, Functions, Objects, TK, Ranges, iterators, arrays, strings.
- Solution to the Java file FAQ - How do I append data to the end of a file using Java? Java file writing FAQ: How do I append text to the end of a text file in Java? The short answer is that you should create a FileWriter instance with the append flag set to true, like.
- Code above has a bug of adding extra new line char at the last line. It should be something like following if(line = reader.readLine())!= null){ stringBuilder.append( line); } while (line = reader.readLine())!= null) { stringBuilder.append( ls); stringBuilder.append.
- I have installed a ruby gem called Redcar, which is launched from the command line. When it runs, it steals the shell until it terminates, so I have to create a new shell window to continue doing. I think this is the easiest and shortest solution to running a batch file.
Let us examine some of the options. Firstly, to get the obvious out of the way. Solaris has a command called mkfile which will allow you to generate a file of a particular size, but we don't have this command on Linux (or at the very least I don't have it on Ubuntu), so I'll leave it at that. If you're on Solaris feel free to investigate. When You Don't Care At All About The Contents Of The File. You just want a file of a particular size, and don't really care what's in it or how many lines it contains – use /dev/zero.
This is a special file on Linux that provides a null character every time you try to read from it. This means we can use it along with the dd command to quickly generate a file of any size. This command will create a file of size count*bs bytes, which in the above case will be 1. Mb. This file will not contain any lines i.
Zambri's answer found here is the best. File.open('out.txt', '<OPTION>') {|f| f.write('write your stuff here') } where your options for <OPTION> are: r - Read only. The file must exist. w - Create an empty file for writing. a - Append to a file.The file is created if it does not.
![Create Or Append To File Ruby Create Or Append To File Ruby](http://i.stack.imgur.com/HLjBu.png)
MB) copied, 0. 0. MB/s. [email protected]: ~/tmp$ ls - al file. The advantages of this approach are as follows: it is blazingly fast taking around 1 second to generate a 1. Gb file (dd if=/dev/zero of=file. Mb)it will create a file of exactly the size that you specified.
This code demonstrates how to create a new XML file using XmlDocument. Here's the XML File: <?xml version='1.0' encoding='utf-8'?> <CategoryList> <Category ID='01'> <MainCategory>XML</MainCategory> <Description>This is a list my XML.
![Create Or Append To File Ruby Create Or Append To File Ruby](https://cdn.tutsplus.com/net/uploads/legacy/822_railsujs/rails3.png)
The disadvantage is the fact that the file will only contain null characters and as a result will not seem to contain any lines. When You Don't Care About The Contents But Want Some Lines.
You want a file of a particular size but don't want it to just be full of nulls, other than that you don't really care. This is a similar case to the above, use /dev/urandom.
This is another special file in Linux, it is a partner of /dev/random which serves as a random number generator on a Linux system. I don't want to go into the mechanics of it, but essentially /dev/random will eventually block unless your system has a lot of activity, /dev/urandom in non- blocking. We don't want blocking when we're creating our files so we use /dev/urandom (the only real difference is that /dev/urandom is actually less random but for our purposes it is random enough : )). The command is similar: dd if=/dev/urandom of=file.
This will create a file with bs*count random bytes, in our case 2. Kb. To generate a 1. Mb file we would do: dd if=/dev/urandom of=file. The file will not contain anything readable, but there will be some newlines in it.[email protected]: ~/tmp$ dd if=/dev/urandom of=file. MB) copied, 1. 3. MB/s. [email protected]: ~/tmp$ wc - l file.
The disadvantages here are the fact that the file does not contain anything readable and the fact that it is quite a bit slower than the /dev/zero method (around 1. Mb). The advantage is the fact that it will contain some lines. You Want Readable Contents But Don't Care If It Is Duplicated. In this case you want to create a file with a particular number of human- readable lines, but you don't really care if the lines are duplicated and don't need the size of the file to be precise. The best way I have found of doing this is as follows.
Here are the specifics. Firstly create a file with two lines in it: cat - > file. This commands redirects STDIN to a file, so you will need to enter two lines and then press Ctrl+D.
Then you will need to run the following command: for i in {1. Where n is an integer.
This will create a file with 2^(n+1) lines in it, by duplicating your original two lines. So to create a file with 1. Here are some more numbers to get you started: n=1. Kb)n=2. 0 will give you 2.
Mb file with 'hello' and 'world' as the two starting lines)n=2. Mb file with 'hello' and 'world' as the two starting lines)Up to n=2.
Here is a handy tip. If you want to quickly empty a file without deleting it, redirect /dev/null into it: cat /dev/null > file. M 2. 01. 0- 0. 3- 2. You Want Readable Contents And No Duplicate Lines. Similar to above, but duplicate lines are an issue for you.
You want a file with a certain number of lines but don't need the size to be precise. In this situation it is a bit of a tall order to do this as a one- liner using pure shell tools (although it is quite possible if you don't mind writing a little script), so we need to turn to one of our beefier friends, Perl or Ruby (whichever one you prefer). I pick Ruby – of course : ).
The idea is as follows. Linux has a dictionary of words which is located at /usr/share/dict/wordswe want to randomly pick a number of words from there to make up into a line then output the linekeep doing this until we get the number of lines we were looking for. The command will look like this: ruby - e 'a=STDIN. X. times do; b=[]; Y. Where X is the number of lines in the file you want to generate and Y is the number of words in each line. So, to create a file with 1.
STDIN. readlines; 1. This is getting a little complex for a one- liner, but once you understand what's going on, it is fairly easy to put together. We basically read in the dictionary into an array, then we randomly select words to form a line (getting rid of newlines while we're at it), we then output our newly created line and we put a loop around the whole thing – pretty simple. With this method you're very unlikely to ever get repeated lines (although it is technically possible). It takes about 1.
Mb file (around 1 million lines with 1. The lines we produce will not make any semantic sense but will be made up of real words and will therefore be readable. There you go, you're now able to generate any kind of data file you want (large or small) for your testing purposes. If you know of any other funky ways to generate files in Linux, then please share and don't forget to grab my RSS feed if you haven't already. Enjoy! Image by MGSpiller.
Ruby File I/O, Directories. Ruby provides a whole set of I/O- related methods implemented in the Kernel module. All the I/O methods are derived from the class IO.
The class IO provides all the basic methods, such as read, write, gets, puts, readline, getc, and printf. This chapter will cover all the basic I/O functions available in Ruby.
For more functions, please refer to Ruby Class IO. The puts Statement: In previous chapters, you assigned values to variables and then printed the output using puts statement.
The puts statement instructs the program to display the value stored in the variable. This will add a new line at the end of each line it writes. Example. #!/usr/bin/ruby. This is variable one". This is variable two".
This will produce the following result. This is variable one. This is variable two. The gets Statement: The gets statement can be used to take any input from the user from standard screen called STDIN. Example: The following code shows you how to use the gets statement.
This code will prompt the user to enter a value, which will be. STDOUT. #!/usr/bin/ruby. Enter a value : ". This will produce the following result.
Enter a value . This is entered value. This is entered value. The putc Statement: Unlike the puts statement, which outputs the entire string onto the screen, the putc statement can be used to output one character at a time. Example: The output of the following code is just the character H.
Hello Ruby!". This will produce the following result. The print Statement: The print statement is similar to the puts statement. The only difference is that the puts statement goes to the next line after printing the contents, whereas with the print statement the cursor is positioned on the same line. Example. #!/usr/bin/ruby.
Hello World". print "Good Morning". This will produce the following result. Hello World. Good Morning. Opening and Closing Files: Until now, you have been reading and writing to the standard input and output. Now, we will see how to play with actual data files.
The File. new Method: You can create a File object using File. Finally, you can use File. Syntax. a. File = File. The File. open Method: You can use File. However, there. is one difference in between File.
File. new methods. The difference is that the File. File. new method.
File. open("filename", "mode") do |a. File|. # .. process the file. Here is a list of The Different Modes of Opening a File: Modes.
Descriptionr. Read- only mode. The file pointer is placed at the beginning of the file. This is the default mode. Read- write mode. The file pointer will be at the beginning of the file.
Write- only mode. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing. Read- write mode. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing. Write- only mode.
The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
Read and write mode. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing. Reading and Writing Files: The same methods that we've been using for 'simple' I/O are available for all file objects. So, gets reads a line from standard input, and a. File. gets reads a line from the file object a.
File. However, I/O objects provides additional set of access methods to make our lives easier. The sysread Method: You can use the method sysread to read the contents of a file. You can open the file in any of the modes when using the method sysread. For example : Following is the input text file.
This is a simple text file for testing purpose. Now let's try to read this file. File = File. new("input.
File. sysread(2. 0). Unable to open file!".
This statement will output the first 2. The file pointer will now be placed at the 2. The syswrite Method: You can use the method syswrite to write the contents into a file. You need to open the file in write mode when using the method syswrite. For example. #!/usr/bin/ruby.
File = File. new("input. File. syswrite("ABCDEF"). Unable to open file!". This statement will write "ABCDEF" into the file. The each_byte Method: This method belongs to the class File.
The method each_byte is always associated with a block. Consider the following code sample. File = File. new("input. File. syswrite("ABCDEF"). File. each_byte {|ch| putc ch; putc ? Unable to open file!".
Characters are passed one by one to the variable ch and then displayed on the screen as follows. The IO. readlines Method: The class File is a subclass of the class IO. The class IO also has some methods, which can be used to manipulate files. One of the IO class methods is IO. This method returns the contents of the file line by line.
The following code displays the use of the method IO. IO. readlines("input. In this code, the variable arr is an array.
Each line of the file input. Therefore, arr[0] will contain the first line, whereas arr[1] will contain the second line of the file. The IO. foreach Method: This method also returns output line by line.
The difference between the method foreach and the method readlines is that the method foreach is associated with a block. However, unlike the method readlines, the method foreach does not return an array. For example. #!/usr/bin/ruby. IO. foreach("input. This code will pass the contents of the file test line by line to the variable block, and then the output will be displayed on the screen. Renaming and Deleting Files: You can rename and delete files programmatically with Ruby with the rename and delete methods. Following is the example to rename an existing file test.
Rename a file from test. File. rename( "test. Following is the example to delete an existing file test. Delete file test. File. delete("test. File Modes and Ownership: Use the chmod method with a mask to change the mode or permissions/access list of a file: Following is the example to change mode of an existing file test.
File. new( "test. Following is the table, which can help you to choose different mask for chmod method: Mask. Description. 07. 00rwx mask for owner. Set user ID on execution. Set group ID on execution. Save swapped text, even after use. File Inquiries: The following command tests whether a file exists before opening it.
File. open("file. File: :exists?( "file. The following command inquire whether the file is really a file. This returns either true or false. File. file?( "text.
The following command finds out if it given file name is a directory. File: :directory?( "/usr/local/bin" ) # => true. File: :directory?( "file. The following command finds whether the file is readable, writable or executable.
File. readable?( "test. File. writable?( "test. File. executable?( "test.
The following command finds whether the file has zero size or not. File. zero?( "test. The following command returns size of the file . File. size?( "text. The following command can be used to find out a type of file .
File: :ftype( "test. The ftype method identifies the type of the file by returning one of the following: file, directory, character. Special. block. Special, fifo, link, socket, or unknown. The following command can be used to find when a file was created, modified, or last accessed . File: :ctime( "test. Fri May 0. 9 1. 0: 0. File: :mtime( "text.
Fri May 0. 9 1. 0: 4. File: :atime( "text. Fri May 0. 9 1. 0: 4. Directories in Ruby: All files are contained within various directories, and Ruby has no problem handling these too. Whereas the File class handles files, directories are handled with the Dir class. Navigating Through Directories: To change directory within a Ruby program, use Dir.
This example changes the current directory to /usr/bin. Dir. chdir("/usr/bin"). You can find out what the current directory is with Dir. Dir. pwd # This will return something like /usr/bin.
You can get a list of the files and directories within a specific directory using Dir. Dir. entries("/usr/bin"). Dir. entries returns an array with all the entries within the specified directory. Dir. foreach provides the same feature. Dir. foreach("/usr/bin") do |entry|. An even more concise way of getting directory listings is by using Dir's class array method. Dir["/usr/bin/*"].
Creating a Directory: The Dir. Dir. mkdir("mynewdir"). You can also set permissions on a new directory (not one that already exists) with mkdir: NOTE: The mask 7. Dir. mkdir( "mynewdir", 7.
Deleting a Directory: The Dir. The Dir. unlink and Dir. Dir. delete("testdir"). Creating Files & Temporary Directories: Temporary files are those that might be created briefly during a program's execution but aren't a permanent store of information. Dir. tmpdir provides the path to the temporary directory on the current system, although the method is not available by default. To make Dir. tmpdir available it's necessary to use require 'tmpdir'.
You can use Dir. tmpdir with File. File. join(Dir. tmpdir, "tingtong").
File. new(tempfilename, "w"). This is a temporary file". File. delete(tempfilename). This code creates a temporary file, writes data to it, and deletes it. Ruby's standard library also includes a library called Tempfile that can create temporary files for you. Tempfile. new('tingtong'). Hello". Built- in Functions: Here is the complete list of ruby built- in functions to process files and directories: Advertisements.