程式扎記

Preface
In this article I explain how to get a list of files inside a folder on the filesystem, a task also called tree traversing, with Go. There are two handy stdlib functions that you can use depending on your goal. I list 3 ways: using filepath.Walk, ioutil.ReadDir or os.File.Readdir.

Using filepath.Walk
The path/filepath stdlib package provides the handy Walk function. It automatically scans subdirectories, though, so make sure this is what you actually want.

Usage is very simple:

view plaincopy to clipboardprint?
package main  
  
import (  
    "fmt"  
    "os"  
    "path/filepath"  
)  
  
func main() {  
    var files []string  
  
    root := "/some/folder/to/scan"  
    err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {  
        files = append(files, path)  
        return nil  
    })  
    if err != nil {  
        panic(err)  
    }  
    for _, file := range files {  
        fmt.Println(file)  
    }  
}  

filepath.Walk accepts a string pointing to the root folder, and a WalkFunc, a function type with signature:

view plaincopy to clipboardprint?
type WalkFunc func(path string, info os.FileInfo, err error) error  

This function is called for each iteration of the folder scan.

The info variable of type os.FileInfo is important because we can get many information on the current file: the name, size, mode, modification time, if it’s a folder or a file, and the underlying data source.

For example we could avoid processing folders by adding:

view plaincopy to clipboardprint?
if info.IsDir() {  
    return nil  
}  

You can exclude (or include) files to the slice based on their extension, by using filepath.Ext and passing the file path:

view plaincopy to clipboardprint?
if filepath.Ext(path) == ".dat" {  
    return nil  
}  

We could store the file name instead of the file path using:

view plaincopy to clipboardprint?
files = append(files, info.Name())  

And we could also define the WalkFunc in a separate closure. We just need to pass a pointer to files in visit:

view plaincopy to clipboardprint?
package main  
  
import (  
    "fmt"  
    "log"  
    "os"  
    "path/filepath"  
)  
  
func visit(files *[]string) filepath.WalkFunc {  
    return func(path string, info os.FileInfo, err error) error {  
        if err != nil {  
            log.Fatal(err)  
        }  
        *files = append(*files, path)  
        return nil  
    }  
}  
  
func main() {  
    var files []string  
  
    root := "/some/folder/to/scan"  
    err := filepath.Walk(root, visit(&files))  
    if err != nil {  
        panic(err)  
    }  
    for _, file := range files {  
        fmt.Println(file)  
    }  
}  

Using ioutil.ReadDir
filepath.Walk is handy but scans subfolders too, by default, which might not be what you want.

The Go stdlib also provides ioutil.ReadDir

view plaincopy to clipboardprint?
func ReadDir(dirname string) ([]os.FileInfo, error)  

ReadDir reads the directory named by dirname and returns a list of directory entries sorted by filename.

and here’s an example from the docs. ioutil.ReadDir takes a folder path as a string and returns a slice of os.FileInfo, which we described above.

view plaincopy to clipboardprint?
package main  
  
import (  
    "fmt"  
    "io/ioutil"  
    "log"  
)  
  
func main() {  
    files, err := ioutil.ReadDir(".")  
    if err != nil {  
        log.Fatal(err)  
    }  
  
    for _, file := range files {  
        fmt.Println(file.Name())  
    }  
}  

Using os.File.Readdir
Internally, ioutil.ReadDir is implemented as:

view plaincopy to clipboardprint?
// ReadDir reads the directory named by dirname and returns  
// a list of directory entries sorted by filename.  
func ReadDir(dirname string) ([]os.FileInfo, error) {  
    f, err := os.Open(dirname)  
    if err != nil {  
        return nil, err  
    }  
    list, err := f.Readdir(-1)  
    f.Close()  
    if err != nil {  
        return nil, err  
    }  
    sort.Slice(list, func(i, j int) bool { return list[i].Name() < list[j].Name() })  
    return list, nil  
}  

as you can see it scans dirname and sorts the files by name. If you don’t need sorting you can as well use:

view plaincopy to clipboardprint?
package main  
  
import (  
    "fmt"  
    "log"  
    "os"  
)  
  
func main() {  
    dirname := "."  
  
    f, err := os.Open(dirname)  
    if err != nil {  
        log.Fatal(err)  
    }  
    files, err := f.Readdir(-1)  
    f.Close()  
    if err != nil {  
        log.Fatal(err)  
    }  
  
    for _, file := range files {  
        fmt.Println(file.Name())  
    }  
}  

Question
I have a problem where I produce a pandas dataframe by concatenating along the row axis (stacking vertically).

Each of the constituent dataframes has an autogenerated index (ascending numbers). After concatenation, my index is screwed up: it counts up to n (where n is the shape[0] of the corresponding dataframe), and restarts at zero at the next dataframe.

I am trying to "re-calculate the index, given the current order", or "re-index" (or so I thought). Turns out that isn't exactly what DataFrame.reindex seems to be doing. Here is what I tried to do:

view plaincopy to clipboardprint?
train_df = pd.concat(train_class_df_list)  
train_df = train_df.reindex(index=[i for i in range(train_df.shape[0])])  

It failed with "cannot reindex from a duplicate axis." I don't want to change the order of my data... just need to delete the old index and set up a new one, with the order of rows preserved.

HowTo
After vertical concatenation, if you get an index of [0, n) followed by [0, m), all you need to do is call reset_index:

view plaincopy to clipboardprint?
train_df.reset_index(drop=True)  

you can do this in place using inplace=True:

view plaincopy to clipboardprint?
>>> import pandas as pd  
>>> cat_df = pd.concat([pd.DataFrame({'a':[1,2]}), pd.DataFrame({'a':[1,2]})])  
>>> cat_df.index  
Int64Index([0, 1, 0, 1], dtype='int64')  
>>> cat_df.reset_index(drop=True, inplace=True)  
>>> cat_df.index  
RangeIndex(start=0, stop=4, step=1)  
>>> cat_df  
   a  
0  1  
1  2  
2  1  
3  2  

程式扎記

標籤

2020年11月6日星期五

[ 文章收集 ] List the files in a folder with Go

[ 常見問題 ] Difference between two time.Time objects

2020年11月1日星期日

[ Python 常見問題 ] Pandas recalculate index after a concatenation

[Git 常見問題] error: The following untracked working tree files would be overwritten by merge

檢舉濫用情形

學習筆記

標籤

2020年11月6日 星期五