Programming in Lua : 21.2.1

This first edition was written for Lua 5.0. While still largely relevant for later versions, there are some differences.
The fourth edition targets Lua 5.3 and is available at Amazon and other bookstores.
By buying the book, you also help to support the Lua project.

Programming in Lua

Part III. The Standard Libraries Chapter 21. The I/O Library

21.2.1 – A Small Performance Trick

Usually, in Lua, it is much faster to read a file as a whole than to read it line by line. However, sometimes we must face some big files (say, tens or hundreds megabytes) for which it is not reasonable to read them all at once. If you want to handle such big files with maximum performance, the fastest way is to read them in reasonably large chunks (e.g., 8 KB each). To avoid the problem of breaking lines in the middle, you simply ask to read a chunk plus a line:

    local lines, rest = f:read(BUFSIZE, "*line")

The variable rest will get the rest of any line broken by the chunk. We then concatenate the chunk and this rest of line. That way, the resulting chunk will always break at line boundaries.

A typical example of that technique is this implementation of wc, a program to count the number of characters, words, and lines in a file:

    local BUFSIZE = 2^13     -- 8K
    local f = io.input(arg[1])   -- open input file
    local cc, lc, wc = 0, 0, 0   -- char, line, and word counts
    while true do
      local lines, rest = f:read(BUFSIZE, "*line")
      if not lines then break end
      if rest then lines = lines .. rest .. '\n' end
      cc = cc + string.len(lines)
      -- count words in the chunk
      local _,t = string.gsub(lines, "%S+", "")
      wc = wc + t
      -- count newlines in the chunk
      _,t = string.gsub(lines, "\n", "\n")
      lc = lc + t
    end
    print(lc, wc, cc)