Releases · go-hep/hep

21 Feb 16:56

sbinet

v0.23.0

398059a

Release v0.23.0: root2fits & fits2root

Release v0.23.0 is fresh from the oven.

This release contains a couple of groot related features and bugfixes.

groot

groot gained 2 new commands:

cmd/root2fits converts a ROOT TTree into a FITS table:

$> root2fits -h
root2fits converts the content of a ROOT tree to a FITS (binary) table.

Usage: root2fits [OPTIONS] -f input.root

Example:

 $> root2fits -f ./input.root -t tree

Options:
  -f string
    	path to input ROOT file name
  -o string
    	path to output FITS file name (default "output.fits")
  -t string
    	name of the ROOT tree to convert

cmd/fits2root converts a FITS table into a ROOT TTree:

$> fits2root -h
fits2root converts the content of a FITS table to a ROOT file and tree.

Usage: fits2root [OPTIONS] -f input.fits

Example:

 $> fits2root -f ./input.fits -t MyHDU

Options:
  -f string
    	path to input FITS file name
  -o string
    	path to output ROOT file name (default "output.root")
  -t string
    	name of the FITS table to convert

Also, many root-xyz commands have been refactored into a simple shim executable that calls into the new groot/rcmd so users (and groot tests) can more easily customize or re-use, say, root-ls, root-dump, ... through an API instead of spawning a sub-process.
This refactor is especially useful for rcmd.Merge, where users can register their own merge strategy for their own types (by implementing the root.Merger interface.)

groot/rtree can now correctly read multi-leaves branches.

And support for TLeafF16, TLeafD32, TProcessID, TRef, TRefArray, TVector2, TVector3, TLorentzVector and TFeldmanCousins has been added.

hplot

fixes for HLine and VLine have been applied: HLine/VLine outside a canvas were incorrectly drawn.

xrootd

migration to gokrb5/v8

ChangeLog

398059a groot/rbase: add initial support for tracking Refs
8c42445 groot: introduce root.UIDer interface
8d26a4a groot{,rdict,rvers}: first stab at TBits
4b94d19 groot{,rdict,rvers}: first stab at TProcessUUID
84a6e03 ci: bump appveyor to Go-1.13
6e0dddd groot{,rdict,rphys,rvers}: add TFeldmanCousins
be0efa3 groot/rphys: properly handle version-2 of TVector{2,3}
c9a6535 groot{,rdict,rphys,rvers}: add TVector2
77598d3 hplot: fix HLine/VLine for out of canvas lines
64086e9 hep: update Go modules
20a7b06 hep,xrootd/xrdproto/auth: migrate to gokrb5/v8
4900ca2 groot: implement TVector3, TLorentzVector
808cc5b groot: add TRefArray
dbf217c groot: add first stab at a TRef implementation
683c737 hbook: fix doc-example of H1D.Integral after Binning1D API migration
c1fb32e cmd/root2arrow: improve automatic installation of arrow-cat
237736a groot/{internal,rdict}: add list-groot-{sinfos,types}
12725a7 groot: generate textual representation of streamers
9ae982c groot/rdict: handling of Long64_t members in ROOT->groot streamer generation
ca69d60 groot/rdict: better handling of groot-wrapped ROOT types in ROOT->groot streamer generation
c19782e groot/rdict: add range-parsing to StreamerElement unmarshaling
499a8f4 groot/riofs: add generation of std::map-based ROOT data tree
2376dc0 groot/riofs/gendata: simplify code generation
d5830ae groot/{rdict,riofs,rtree}: improve STL-container name arguments parsing
7118537 groot/{rdict,rtree}: rename StreamerSTL.STLVectorType into STLType
8e85fb9 xroot/xrdproto/auth/krb5: migrate to gokrb5.v7
a44b1ea hep: go mod tidy
3216604 hplot: make sure embedmd is installed+required
fb7f30f hep: remove link to opencollective
64d644a groot: implement r/w TLeafF16
d17364c groot: implement r/w TLeafD32
c9904cb groot/rdict: add support for TLeaf{D32,F16} comment/range parsing
98e75b5 groot{,/riofs,/rvers}: add support for TLeafF16 and TLeafD32
5621593 groot/{cmd/root-diff,rcmd}: refactor root-diff into rcmd.Diff
4c1484a groot/rcmd: refactor Dump into a dumpCmd struct
9e53d27 groot/{cmd/root-cp,rcmd}: export root-cp as rcmd.Copy
a1734ed groot/{cmd/root-merge,rcmd}: export root-merge as rcmd.Merge
6851230 cmd/arrow2root: use rcmd.Dump
944c4a9 cmd/fits2root: use groot/rcmd.Dump in tests
e20ccab groot: export rcmd package
098e848 groot: more pervasive use of rtests.RunCxxROOT
af6ef10 groot/rtree: handle merging of Atlas flat-tuples
1df725b groot/riofs: add support for TProcessID StreamerInfo
0e1f43f groot/{rbase,rtypes}: implement TProcessID
1978a74 groot{,/rvers}: add support for TProcessID
ebbd064 groot/rdict: better error message
d3b1b41 groot/rmeta: better error message
cee0755 groot/rtree: add more Branch.setAddress tests
aad9221 groot/riofs: use rtests for generating testdata
98fe128 groot/{internal/rcmd,riofs}: add testfile w/ & w/o padding
4419621 groot/internal/rcmd: properly display branches w/ multi-leaves in root-dump
e25f2ad groot/rtree: support reading multi-leaves branches
3c58df0 cmd/root2fits: first import
f15977d cmd/fits2root: first import
3e92a08 all: 2020 is the year of the Gopher

Assets 2

20 Dec 16:13

sbinet

v0.22.0

8a851ee

Release v0.22.0: root-merge and Happy New Year

Release v0.22.0 is fresh from the oven.

This release contains a couple of groot related features and bugfixes.

groot

Building on the work on tree-writing from v0.21.00, groot gained 3 new commands:

cmd/arrow2root converts an ARROW data file into a corresponding ROOT tree,

$> arrow2root testdata/primitives.file.data
$> root-ls -t output.root
=== [./output.root] ===
version: 61804
  TTree      tree         tree    (entries=10)
    bools    "bools/O"    TBranch
    int8s    "int8s/B"    TBranch
    int16s   "int16s/S"   TBranch
    int32s   "int32s/I"   TBranch
    int64s   "int64s/L"   TBranch
    uint8s   "uint8s/b"   TBranch
    uint16s  "uint16s/s"  TBranch
    uint32s  "uint32s/i"  TBranch
    uint64s  "uint64s/l"  TBranch
    float32s "float32s/F" TBranch
    float64s "float64s/D" TBranch

cmd/npy2root converts a NumPy data file into a correspondig ROOT tree,

$> npy2root -h
npy2root converts the content of a NumPy data file to a ROOT file and tree.

Usage: npy2root [OPTIONS] input.npy

The NumPy data file format is described here:

 https://docs.scipy.org/doc/numpy/neps/npy-format.html

Example:

 $> npyio-ls input.npy
 ================================================================================
 file: input.npy
 npy-header: Header{Major:1, Minor:0, Descr:{Type:<f8, Fortran:false, Shape:[2 3]}}
 data = [0 1 2 3 4 5]

 $> npy2root -o output.root -t mytree ./input.npy
 $> root-ls -t ./output.root
 === [./output.root] ===
 version: 61804
   TTree   mytree       mytree  (entries=2)
     numpy "numpy[3]/D" TBranch

 $> root-dump ./output.root
 >>> file[./output.root]
 key[000]: mytree;1 "mytree" (TTree)
 [000][numpy]: [0 1 2]
 [001][numpy]: [3 4 5]

Options:
  -o string
    	path to output ROOT file (default "output.root")
  -t string
    	name of the output ROOT tree (default "tree")

groot/cmd/root-merge is the beginning of an hadd-like command, merging (for now) trees and graphs.

$> root-merge -h
Usage: root-merge [options] file1.root [file2.root [file3.root [...]]]

ex:
 $> root-merge -o out.root ./testdata/chain.flat.1.root ./testdata/chain.flat.2.root

options:
  -o string
    	path to merged output ROOT file (default "out.root")
  -v	enable verbose mode

$> root-merge ./testdata/chain.flat.*
$> root-ls -t ./out.root 
=== [./out.root] ===
version: 61804
  TTree    tree           my tree title (entries=10)
    B      "B/O"          TBranch
    Str    "Str/C"        TBranch
    I8     "I8/B"         TBranch
    I16    "I16/S"        TBranch
    I32    "I32/I"        TBranch
    I64    "I64/L"        TBranch
    U8     "U8/b"         TBranch
    U16    "U16/s"        TBranch
    U32    "U32/i"        TBranch
    U64    "U64/l"        TBranch
    F32    "F32/F"        TBranch
    F64    "F64/D"        TBranch
    ArrBs  "ArrBs[10]/O"  TBranch
    ArrI8  "ArrI8[10]/B"  TBranch
    ArrI16 "ArrI16[10]/S" TBranch
    ArrI32 "ArrI32[10]/I" TBranch
    ArrI64 "ArrI64[10]/L" TBranch
    ArrU8  "ArrU8[10]/b"  TBranch
    ArrU16 "ArrU16[10]/s" TBranch
    ArrU32 "ArrU32[10]/i" TBranch
    ArrU64 "ArrU64[10]/l" TBranch
    ArrF32 "ArrF32[10]/F" TBranch
    ArrF64 "ArrF64[10]/D" TBranch
    N      "N/I"          TBranch
    SliBs  "SliBs[N]/O"   TBranch
    SliI8  "SliI8[N]/B"   TBranch
    SliI16 "SliI16[N]/S"  TBranch
    SliI32 "SliI32[N]/I"  TBranch
    SliI64 "SliI64[N]/L"  TBranch
    SliU8  "SliU8[N]/b"   TBranch
    SliU16 "SliU16[N]/s"  TBranch
    SliU32 "SliU32[N]/i"  TBranch
    SliU64 "SliU64[N]/l"  TBranch
    SliF32 "SliF32[N]/F"  TBranch
    SliF64 "SliF64[N]/D"  TBranch

groot/rtree also saw some activity:

fix for correctly writing []int8 and []uint8 branch data
fix for correctly writing empty strings (subtle bug when the empty string was the first or the last to be written)
fix for correctly writing trees under (deeply) nested directories
groot/rtree.Copy to easily copy trees from one place to another (possibly across files)
support for writing leaves with n-dim arrays (n>=2)

The root-print, root-cp and root-merge commands have been improved to correctly handle (arbitrarily) nested directories.

groot gained support for compressing and decompressing data via the Zstandard library.
ROOT-6.20/00 should be released with this feature as well.
While working on the rcompress package (that handles all compression/decompression aspects of groot), a bit of optimization work has been carried out, resulting in ~10% optimization in compression speed and a ~40% improvement in memory usage during compression.

hbook

hbook gained 2 new APIs:

hbook.H1D#FillN that fills a 1D-histogram with a slice of data and its associated weights
hbook.H12D#FillN that fills a 2D-histogram with a slice of data and its associated weights

FUSE

Support for mounting ROOT files and remote XRootD location thru FUSE has been removed from go-hep.org/x/hep.
FUSE support on macOS is not open-source anymore and the provided feature thru Go-HEP was a bit flaky (at least, the tests were flaky).
The xrdfuse package, xrd-fuse and root-fuse commands have been migrated to go-hep.org/x/exp.

We might consider using the 9p protocol instead of FUSE.

AOB

See you in 2020!

ChangeLog

8a851ee all: update URL of NumPy array data file specification
934bdd7 groot/internal/rcompress: optimize compression
5b1a2dd groot/internal/rcompress: fix roundtrip compression
bd257a1 groot/internal/rcompress: add compress benchmark
5801a56 hep: update go.mod
08b1f75 groot/riofs: handle riofs.Dir in riofs.FileOf
c5200f1 groot/cmd/root-merge: first import
590a5c3 groot/rtree: make Writer implement root.Merger
dfecc37 groot/rhist: implement root.Merger interface for Graph{,Errors}
28fe971 groot/root: introduce Merger interface
1437ad2 groot,xrootd: move FUSE related code to go-hep.org/x/exp
d45349f groot/internal/rcompress: streamline error strings
1005829 groot/internal/rcompress: use rtests.RunCxxROOT to run ROOT macros
54eedba groot/internal/rcompress: add support for flate.Best{Compression,Speed} to Zstd
7c7bdf3 groot/{riofs,internal/rcompress}: add support for zstd (de)compressor
1d81a65 groot/cmd/root-print: add test for '^dir'
4f47f41 groot/cmd/root-print: support nested-dirs
9db9056 groot/cmd/root-cp: add support for nested-directories
5ee79ea groot/cmd/root-cp: add support for copying trees
11df8e6 cmd/npy2root: add support for n-dim arrays
8b51e1e groot/rtree: add support for leaves with n-dim arrays (n>=2)
049273a cmd/npy2root: first import
45e282c hbook: add H2D.FillN(xs, ys, ws)
4811e2c hbook: add H1D.FillN(xs, ws)
241c5af cmd/arrow2root,groot/rarrow: introduce rarrow.NewFlatTreeWriter
64dd534 cmd/arrow2root: first import
6ecf64a groot/rtree: introduce Copy and CopyN for copying trees
c8dbaf0 groot: introduce groot/internal/rcmd
5c5a8e9 groot/rtree: test writing []int8 and [N]int8
2f7935b groot/rtree: support writing trees w/ empty strings
3a56e78 groot/rbytes: make WriteString correctly handle empty strings as first value
ebe3617 groot/{riofs,rtree}: add support for writing n-tuples under nested directories

Assets 2

28 Nov 16:18

sbinet

v0.21.0

a3bdc48

Release v0.21.0: writing trees -- with slices and from structs

Release v0.21.0 is fresh from the oven.

This release contains a major new groot feature: the ability to write (simple, flat) trees, including variable-length arrays (a.k.a slices):

groot

groot supports more ROOT-4 files (as created by Geant4: TH{1,2}x and TTree)
fixed compilation on 32b systems
add groot/rtree.WriterVarsFromStruct to generate a slice of rtree.WriterVars from a user-provided struct that can be then used to fill a tree
add With{LZ4,LZMA,Zlib} and WithoutCompression function to configure whether a tree should use compression (and what kind of compression, if any)
add WithBasketSize to configure the basket size of trees/branches
add auto-flushing of branches' baskets

godoc:example

type Data struct {
    I32    int32
    F64    float64
    Str    string
    ArrF64 [5]float64
    N      int32
    SliF64 []float64 `groot:"SliF64[N]"` // tell ROOT/C++ the leaf name and the leaf holding the count
}

const (
    fname = "struct.root"
    nevts = 5
)

f, err := groot.Create(fname)
if err != nil {
    log.Fatalf("%+v", err)
}
defer f.Close()

var evt Data

tree, err := rtree.NewWriter(f, "mytree", rtree.WriteVarsFromStruct(&evt))
if err != nil {
    log.Fatalf("could not create tree writer: %+v", err)
}

fmt.Printf("-- created tree %q:\n", tree.Name())
for i, b := range tree.Branches() {
    fmt.Printf("branch[%d]: name=%q, title=%q\n", i, b.Name(), b.Title())
}

for i := 0; i < nevts; i++ {
    evt.I32 = int32(i)
    evt.F64 = float64(i)
    evt.Str = fmt.Sprintf("evt-%0d", i)
    evt.ArrF64 = [5]float64{float64(i), float64(i + 1), float64(i + 2), float64(i + 3), float64(i + 4)}
    evt.N = int32(i)
    evt.SliF64 = []float64{float64(i), float64(i + 1), float64(i + 2), float64(i + 3), float64(i + 4)}[:i]
    _, err = tree.Write()
    if err != nil {
        log.Fatalf("could not write event %d: %+v", i, err)
    }
}
fmt.Printf("-- filled tree with %d entries\n", tree.Entries())

err = tree.Close()
if err != nil {
    log.Fatalf("could not write tree: %+v", err)
}

err = f.Close()
if err != nil {
    log.Fatalf("could not close tree: %+v", err)
}

// Output:
// -- created tree "mytree":
// branch[0]: name="I32", title="I32/I"
// branch[1]: name="F64", title="F64/D"
// branch[2]: name="Str", title="Str/C"
// branch[3]: name="ArrF64", title="ArrF64[5]/D"
// branch[4]: name="N", title="N/I"
// branch[5]: name="SliF64", title="SliF64[N]/D"
// -- filled tree with 5 entries
// -- read back ROOT file
// entry[0]: {I32:0 F64:0 Str:evt-0 ArrF64:[0 1 2 3 4] N:0 SliF64:[]}
// entry[1]: {I32:1 F64:1 Str:evt-1 ArrF64:[1 2 3 4 5] N:1 SliF64:[1]}
// entry[2]: {I32:2 F64:2 Str:evt-2 ArrF64:[2 3 4 5 6] N:2 SliF64:[2 3]}
// entry[3]: {I32:3 F64:3 Str:evt-3 ArrF64:[3 4 5 6 7] N:3 SliF64:[3 4 5]}
// entry[4]: {I32:4 F64:4 Str:evt-4 ArrF64:[4 5 6 7 8] N:4 SliF64:[4 5 6 7]}

hplot

hplot.Function has been copied from Gonum/plot with extra support for log-y axes (automatically discarding intervals where the returned value is invalid on log-y axes)

godoc:example

quad := hplot.NewFunction(func(x float64) float64 { return x * x })
quad.Color = color.RGBA{B: 255, A: 255}

fun := hplot.NewFunction(func(x float64) float64 {
    switch {
    case x < 6:
        return 20
    case 6 <= x && x < 7:
        return 0
    case 7 <= x && x < 7.5:
        return 30
    case 7.5 <= x && x < 9:
        return 0
    case 9 <= x:
        return 40
    }
    return 50
})
fun.LogY = true
fun.Color = color.RGBA{R: 255, A: 255}

p := hplot.New()
p.Title.Text = "Functions - Log-Y scale"
p.X.Label.Text = "X"
p.Y.Label.Text = "Y"

p.Y.Scale = plot.LogScale{}
p.Y.Tick.Marker = plot.LogTicks{}

p.Add(fun)
p.Add(quad)
p.Add(hplot.NewGrid())
p.Legend.Add("x^2", quad)
p.Legend.Add("fct", fun)
p.Legend.ThumbnailWidth = 0.5 * vg.Inch

p.X.Min = 5
p.X.Max = 10
p.Y.Min = 10
p.Y.Max = 100

err := p.Save(200, 200, "testdata/functions_logy.png")
if err != nil {
    log.Panic(err)
}

clean-up

The whole GoHEP tree has been cleaned up to remove the use of github.com/pkg/errors and use instead x/xerrors.
x/xerrors will be phased out (in favor of fmt from the stdlib) when Go-1.13 will be the oldest supported release.

This is to gain nice error reports as documented in the Go blog:

https://blog.golang.org/go1.13-errors

AOB

The GoHEP website has also gained a new /dist section, where binaries for Darwin, Freebsd, Linux and Windows are uploaded (for every new tagged release).

We'll try to release v0.22.0 with the ability to write structs and nested trees.

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

ChangeLog

a3bdc48 groot/rhist: add test for ROOT-4 H1D
6c8c87a groot: add support for ROOT-4 histograms
c5fa1b7 groot/rtree: support G4-like tree
508bea9 groot/rtree: TBranch-v6
2afe0e7 groot/rtree: TBranchElement-v1
96ca4b3 groot/rtree: TTree-v5
d1049cd groot/rtree: TBranch-v10
f3ad4a5 hplot: add support for log-y based functions
0b630d8 ci: add x-compile release script
0fa7206 ci: display C++ ROOT version
c79f47a groot/{riofs,rtree}: fix 32b build
b2bb9f0 ci: test 32b builds
5f73852 groot/internal/rtests: add ROOTError wrap test
ac4885b ci: add binary ROOT installation scripts
91c9502 groot/rtree: add WriterVarsFromStruct helper function
7379a85 groot/rtree: make NewScanVars support Slices and Arrays
9be2300 groot/rtree: test WithXXX API and ability to correctly read back trees from C++ ROOT
a1a9024 groot/internal/rtests: introduce RunCxxROOT to (more easily) run ROOT macros
ac1dc83 groot/{riofs,internal/rcompress,rtree}: introduce rcompress.Settings
f35da3b groot/{riofs,rtree}: add WithXYZ API to configure how trees should be created
813c909 ci: add mk-release script to push versions to index.golang
d12963d fwk: apply golangci suggestions
295baa5 fwk: simplify fowk.MsgStream interface
86f5de3 all: use x/xerrors
f9cbd76 groot/rtree: improve diagnostic+doc for rtree.New(Tree)Scanner
c4a04e9 groot/rarrow: fix Arrow-struct creation data-offset
b8a39b7 groot/rtree: handle auto-flush
b545af5 groot/rtree: add support var-len arrays to trees

Assets 2

13 Nov 18:14

sbinet

v0.20.0

6368b7c

Release v0.20.0: writing trees

Release v0.20.0 is fresh from the oven.

This release contains a major new groot feature: the ability to write (simple, flat) trees:

groot/rtree#NewWriter

and a couple of new features for hplot.

groot

groot support has been bumped to ROOT-6.18/04
groot can now create nested directories that are compatible with ROOT.
Previously, groot would create nested directories which could only be read back by groot (and uproot).

w, err := groot.Create("subdirs.root")
if err != nil {
	log.Fatal(err)
}
defer w.Close()

dir1, err := w.Mkdir("dir1")
if err != nil {
	log.Fatal(err)
}

dir11, err := dir1.Mkdir("dir11")
if err != nil {
	log.Fatal(err)
}

err = dir11.Put("obj1", rbase.NewObjString("data-obj1"))
if err != nil {
	log.Fatal(err)
}

dir2, err := w.Mkdir("dir2")
if err != nil {
	log.Fatal(err)
}

err = dir2.Put("obj2", rbase.NewObjString("data-obj2"))
if err != nil {
	log.Fatal(err)
}

err = w.Close()
if err != nil {
	log.Fatal(err)
}

// Output:
// >> subdirs.root
// >> subdirs.root/dir1
// >> subdirs.root/dir1/dir11
// >> subdirs.root/dir1/dir11/obj1
// >> subdirs.root/dir2
// >> subdirs.root/dir2/obj2

cmd/root2csv can now handle TLeafElement leaves of builtins
groot/riofs.File.SegmentMap has been added to riofs.File to inspect the list of ROOT records in a file
support for writing simple flat trees of builtins and arrays of builtins (no slices, yet) has been added:

godoc:example

type Data struct {
	I32    int32
	F64    float64
	Str    string
	ArrF64 [5]float64
}
const (
	fname = "groot-flat-ntuple.root"
	nevts = 5
)

f, err := groot.Create(fname)
if err != nil {
	log.Fatalf("%+v", err)
}
defer f.Close()

var evt Data

wvars := []rtree.WriteVar{
	{Name: "I32", Value: &evt.I32},
	{Name: "F64", Value: &evt.F64},
	{Name: "Str", Value: &evt.Str},
	{Name: "ArrF64", Value: &evt.ArrF64},
}
tree, err := rtree.NewWriter(f, "mytree", wvars)
if err != nil {
	log.Fatalf("could not create tree writer: %+v", err)
}

fmt.Printf("-- created tree %q:\n", tree.Name())
for i, b := range tree.Branches() {
	fmt.Printf("branch[%d]: name=%q, title=%q\n", i, b.Name(), b.Title())
}

for i := 0; i < nevts; i++ {
	evt.I32 = int32(i)
	evt.F64 = float64(i)
	evt.Str = fmt.Sprintf("evt-%0d", i)
	evt.ArrF64 = [5]float64{float64(i), float64(i + 1), float64(i + 2), float64(i + 3), float64(i + 4)}
	_, err = tree.Write()
	if err != nil {
		log.Fatalf("could not write event %d: %+v", i, err)
	}
}
fmt.Printf("-- filled tree with %d entries\n", tree.Entries())

err = tree.Close()
if err != nil {
	log.Fatalf("could not write tree: %+v", err)
}

err = f.Close()
if err != nil {
	log.Fatalf("could not close tree: %+v", err)
}

// Output:
// -- created tree "mytree":
// branch[0]: name="I32", title="I32/I"
// branch[1]: name="F64", title="F64/D"
// branch[2]: name="Str", title="Str/C"
// branch[3]: name="ArrF64", title="ArrF64[5]/D"
// -- filled tree with 5 entries
// -- read back ROOT file
// entry[0]: {I32:0 F64:0 Str:evt-0 ArrF64:[0 1 2 3 4]}
// entry[1]: {I32:1 F64:1 Str:evt-1 ArrF64:[1 2 3 4 5]}
// entry[2]: {I32:2 F64:2 Str:evt-2 ArrF64:[2 3 4 5 6]}
// entry[3]: {I32:3 F64:3 Str:evt-3 ArrF64:[3 4 5 6 7]}
// entry[4]: {I32:4 F64:4 Str:evt-4 ArrF64:[4 5 6 7 8]}

hplot

hplot.VLine has been added to easily create a vertical line on a plot
hplot.HLine has been added to easily create a horitzontal line on a plot
hplot.NewBand has been added to easily create a band between two sets of data points on a plot

AOB

We'll try to release v0.21.0 shortly with the ability to write slices to a tree and perhaps structs.

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Assets 2

31 May 17:54

sbinet

v0.19.0

3de0699

Release v0.19.0

Release v0.19.0 is fresh from the oven.

This release contains 3 new groot-related packages:

groot/rsql/rsqldrv: a package to present a ROOT File+Tree as a SQL database, by way of implementing the database/sql/driver interface for groot
groot/rsql: a convenience package to scan (a la TTree::Scan and TTree::Draw) rtree.Tree and create hbook.H1D or hbook.H2D
groot/rarrow: a package to present rtree.Tree as Apache Arrow's Tables and Records. This will be very useful for interoperability with DataScience and/or Machine Learning toolkits.

groot

Providing groot/rarrow made a new command possible - cmd/root2arrow - that can convert a ROOT TTree stored in a ROOT file, into a sequence of Arrow records, stored into an Arrow file (or Arrow stream.)

$> go doc go-hep.org/x/hep/cmd/root2arrow
root2arrow converts the content of a ROOT TTree to an ARROW file.

    Usage of root2arrow:
      -o string
        	path to output ARROW file name (default "output.data")
      -stream
        	enable ARROW stream (default is to create an ARROW file)
      -t string
        	name of the tree to convert (default "tree")

    $> root2arrow -o foo.data -t tree ../../groot/testdata/simple.root
    $> arrow-ls ./foo.data
    version: V4
    schema:
      fields: 3
        - one: type=int32
        - two: type=float32
        - three: type=utf8
    records: 1
    $> arrow-cat ./foo.data
    version: V4
    record 1/1...
      col[0] "one": [1 2 3 4]
      col[1] "two": [1.1 2.2 3.3 4.4]
      col[2] "three": ["uno" "dos" "tres" "quatro"]

Here is an example of rsql.Scan:

func main() {
	f, err := groot.Open("../testdata/simple.root")
	if err != nil {
	    log.Fatal(err)
	}
	defer f.Close()
	
	o, err := f.Get("tree")
	if err != nil {
	    log.Fatal(err)
	}
	
	tree := o.(rtree.Tree)
	
	var (
	    v1s []int32
	    v2s []float64
	    v3s []string
	)
	
	err = rsql.Scan(tree, "SELECT (one, two, three) FROM tree", func(x int32, y float64, z string) error {
	    v1s = append(v1s, x)
	    v2s = append(v2s, y)
	    v3s = append(v3s, z)
	    return nil
	})
	
	if err != nil {
	    log.Fatal(err)
	}
	
	fmt.Printf("tree[%q]: %v\n", "one", v1s)   // tree["one"]: [1 2 3 4]
	fmt.Printf("tree[%q]: %v\n", "two", v2s)   // tree["two"]: [1.1 2.2 3.3 4.4]
	fmt.Printf("tree[%q]: %q\n", "three", v3s) // tree["three"]: ["uno" "dos" "tres" "quatro"]
}

And one example of rsql.ScanH1D:

func main() {
	f, err := groot.Open("../testdata/simple.root")
	if err != nil {
	    log.Fatal(err)
	}
	defer f.Close()
	
	o, err := f.Get("tree")
	if err != nil {
	    log.Fatal(err)
	}
	
	tree := o.(rtree.Tree)
	
	h, err := rsql.ScanH1D(tree, "SELECT two FROM tree", nil)
	if err != nil {
	    log.Fatal(err)
	}
	
	fmt.Printf("entries: %v\n", h.Entries()) // entries: 4
	fmt.Printf("x-mean: %v\n", h.XMean())    // x-mean: 2.75
}

AOB

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Assets 2

31 May 17:30

sbinet

v0.18.0

d772a7d

Release v0.18.0

Release v0.18.0 is fresh-ish from the oven.

This release contains the beginning of an XRootD server in Go (thanks Mikhail!) as well as yet more groot improvements.

fit

fit: update for new gonum.org/v1/gonum/optimize API
fit: use new gonum/{diff/fd,optimize} API

groot

groot/riofs.Open and thus groot.Open have been modified to only open local files by default.
A "plugin" mechanism has been devised to be able to optionally install plugins to deal with, e.g., HTTP-served and XRootD-served files.
To enable these plugins, one just need to "blank import" the following packages:

import (
	_ "go-hep.org/x/hep/groot/riofs/plugin/http"
	_ "go-hep.org/x/hep/groot/riofs/plugin/xrootd"
)

The groot/cmd/root-xxx commands have been modified to automatically import the above packages.
Users are expected to do the same if they want to transparently be able to read such files.
The rationale being that groot/riofs should by default try to always create static binaries: to read the HTTP- and XRootD-served files one needs to import net/http that require some amount of dynamic linking.

The groot package also gained a new command:

$> root-gen-type -h
Usage: root-gen-type [options] input.root

ex:
 $> root-gen-type -p mypkg -t MyType -o streamers_gen.go ./input.root

options:
  -o string
    	output file name
  -p string
    	package import path
  -t string
    	comma-separated list of (regexp) type names (default ".*")
  -v	enable verbose mode

root-gen-type automatically creates the streamer code from an input ROOT file (not unlike the old genreflex or the new rootcling) as well as a Go struct compatible with the data contained in the (C++, on-disk) streamers (kind of like the C++ TTree::MakeClass.)

groot: add support for TMap
groot/rdict: add support for StreamerObject{,Pointer}
groot/cmd/root-ls: lazily load data off disk
groot/rvers: migrate to ROOT-6.16/00
groot: add support for TClonesArray
groot: introduce ROOT open plugin mechanism
groot/cmd/root-gen-type: first import
groot: support TArrayC, TArrayS and TArrayL
groot/rdict: add support for slices of Object-s or ObjectAny-s
groot: add initial support for Float16_t and Double32_t in root-gen-type

hbook

hbook/ntup: add an example on how to create histos from CSV
hbook/ntup: add examples for Scan and ScanH1D methods

hep

hep: add Version to API

hplot

hplot: add support for H1D-logy

xrootd

xrootd: add ping, mkdir, rm, and rmdir support to the server
xrootd/cmd/xrd-ls: fix output format
xrootd/xrdproto/query: update example for new XRootD version

AOB

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Assets 2

15 Jan 11:26

sbinet

v0.17.1

b0a13fa

Release v0.17.1

Release v0.17.1 is fresh-ish from the oven.

This new release continues the refactoring and consolidation work of the groot package (meant to replace rootio.)

brio

brio/cmd/brio-gen gained some documentation

groot

groot/rdict has seen a bunch of work to create user-type StreamerInfos,
groot/cmd/root-gen-streamer: new command to automatically generate a StreamerInfo and its StreamerElements given a ROOT or user type,
groot/rdict now properly handles streamers with arrays of builtins, and properly visits std::vector<T> fields (where T is a builtin)
groot/cmd/root-dump: now properly handle root.List values
groot dropped the use of gofrs/uuid and replaced it with hashicorp/go-uuid. gofrs/uuid dropped support for Go modules and broke the build of Go-HEP.

Here is usage example of the new root-gen-streamer command:

$> root-gen-streamer -help
Usage: root-gen-streamer [options]

ex:
 $> root-gen-streamer -p image -t Point -o streamers_gen.go
 $> root-gen-streamer -p go-hep.org/x/hep/hbook -t Dist0D,Dist1D,Dist2D -o foo_streamer_gen.go

options:
  -o string
    	output file name
  -p string
    	package import path
  -t string
    	comma-separated list of type names
  -v	enable verbose mode

xrootd

The xrootd/client and xrootd/server packages have been merged into a single package, xrootd.
This simplified the import structure as well as reduced the amount of boilerplate code that was duplicated between the two packages.

For users of xrootd/client:

diff --git a/xrootd/cmd/xrd-ls/main.go b/xrootd/cmd/xrd-ls/main.go
index 784a5c7..9961802 100644
--- a/xrootd/cmd/xrd-ls/main.go
+++ b/xrootd/cmd/xrd-ls/main.go
@@ -31,7 +31,7 @@ import (
        "text/tabwriter"
 
        "github.com/pkg/errors"
-       xrdclient "go-hep.org/x/hep/xrootd/client"
+       "go-hep.org/x/hep/xrootd"
        "go-hep.org/x/hep/xrootd/xrdfs"
        "go-hep.org/x/hep/xrootd/xrdio"
 )
@@ -93,7 +93,7 @@ func xrdls(name string, long, recursive bool) error {
 
        ctx := context.Background()
 
-       c, err := xrdclient.NewClient(ctx, url.Addr, url.User)
+       c, err := xrootd.NewClient(ctx, url.Addr, url.User)
        if err != nil {
                return errors.Errorf("could not create client: %v", err)
        }

For users of xrootd/server:

diff --git a/xrootd/cmd/xrd-srv/main.go b/xrootd/cmd/xrd-srv/main.go
index 2a09dbd..230bf31 100644
--- a/xrootd/cmd/xrd-srv/main.go
+++ b/xrootd/cmd/xrd-srv/main.go
@@ -14,7 +14,7 @@ import (
        "os"
        "os/signal"
 
-       "go-hep.org/x/hep/xrootd/server"
+       "go-hep.org/x/hep/xrootd"
 )
 
 func init() {
@@ -56,7 +56,7 @@ func main() {
                log.Fatalf("could not listen on %q: %v", *addr, err)
        }
 
-       srv := server.New(server.NewFSHandler(baseDir), func(err error) {
+       srv := xrootd.NewServer(xrootd.NewFSHandler(baseDir), func(err error) {
                log.Printf("an error occured: %v", err)
        })

AOB

Support for writing TTrees didn't make it under the X-mas tree.
This has been converted to a NYE resolution, though.

Before tackling this big item, support for reading TClonesArray is on the way (and tracked in the sbinet/hep#issue-419 branch.)

A proposal for data frames is in the works in the gonum/exp repository.
Feel free to comment on the associated pull request (#19) or on the gonum-dev forum.

Finally, this is GSoC proposal season.
I (@sbinet) will probably send a GDML oriented proposal.
Feel free to send or discuss yours on the go-hep mailing list.

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Assets 2

03 Dec 13:09

sbinet

v0.16.0

105cc3b

Release v0.16.0

Release v0.16.0 is fresh from the oven.

This new release introduces a new package, groot, that replaces rootio.

groot & rootio

groot is a new package that is meant to replace rootio.
For backward compatibility, rootio will remain in the repository for a couple of releases (presumably until v1.0.0.)

groot is the result of refactoring rootio in a couple of more focused packages with clear API boundaries:

groot/rbase: definitions of basic ROOT classes (Object, Named, ObjString, ...)
groot/rbytes: definitions of types useful for serializing and deserializing ROOT data buffers, interfaces to interact with ROOT's metadata classes such as StreamerInfo and StreamerElements.
groot/rcont: definitions of ROOT container types (TList, THashList, TObjArray, TArrayX)
groot/rdict: definitions of ROOT streamers (TStreamerArtificial, TStreamerLoop, ...)
groot/rhist: definitions of ROOT types related to histograms and graphs (TH1x, TH2x, TGraph, TGraphErrors, TGraphAsymmErrors)
groot/riofs: low-level types and functions to deal with opening and creating ROOT files; users should prefer using the groot package to open and create ROOT files
groot/root: ROOT core interfaces (Object, Named, ObjArray, ...)
groot/rsrv: exposes HTTP end-point to manipulate ROOT files, plot and create histograms and graphs from files or trees
groot/rtree: interface to decode, read, concatenate and iterate over ROOT Trees
groot/rtypes: rtypes contains the means to register types (ROOT ones and user defined ones) with the ROOT type factory system
groot/rvers: rvers contains the ROOT version and the classes' versions groot is supporting and currently reading.

Interacting with ROOT files should be performed with the groot package:

import "go-hep.org/x/hep/groot"

func F() {
	f1, err := groot.Open("some/file.root")
	f2, err := groot.Create("some/other.root")
}

groot has the needed bootstrap code that allows to automatically import the registration code for all the ROOT types groot knows how to handle.

Here is a quick and dirty Rosetta code for migrating to groot:

rootio.H1 → groot/rhist.H1
rootio.H2 → groot/rhist.H2
rootio.Graph → groot/rhist.Graph
rootio.Tree → groot/rtree.Tree
rootio.ChainOf → groot/rtree.ChainOf
rootio.Scanner → groot/rtree.Scanner
rootio.File → groot/riofs.File
rootio.Directory → groot/riofs.Directory
rootio.Open → groot.Open
rootio.Create → groot.Create

groot/rsrv & root-srv

root-srv has been refactored to extract the pure plot/file interaction machinery from the GUI part.
The plot creation and the ROOT file interaction parts have been refactored into a new package go-hep.org/x/hep/groot/rsrv that contains a couple of HTTP end-points that can be reused in third-party packages or applications.

rsrv exposes a REST API that expects JSON requests (OpenFileRequest, PlotH1Request, PlotTreeRequest, ...) and returns JSON responses.
The HTTP end-points are attached to the rsrv.Server type.

root-srv can now open ROOT files served over xrootd.

hbook

hbook now exposes a Bin method on H{1,2}D to retrieve a bin by its (x,y) coordinates.

hplot

Following an issue raised on gonum/plot, hplot now creates histograms with transparent background by default.

rio

Improved test coverage (by adding some more tests.)

xrootd

The low-level bits for the following requests have been implemented:

kXR_query
kXR_prepare
kXR_endsess
kXR_locate
kXR_decrypt
kXR_admin

The first steps to support the "host" security provider have also been implemented.

We are still missing the implementation for the GSI authentification protocol: still waiting on xrootd to provide specifications for this protocol (progress is tracked here: issue-757.)

Improved test coverage.

AOB

We will try to have preliminary support for writing TTrees in the next release.
That should be fun.

Interoperability with Apache Arrow Arrays is in the works.
It might even prove to be easier to support Apache Arrow first and then implement TTrees writing support on top of that.
We will see...

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Assets 2

06 Sep 13:41

sbinet

v0.15.0

46d80ef

Release v0.15.0

Release v0.15.0 is fresh from the oven.

This new release dropped explicit support for Go-1.8.x but, in turn, gained support for Go-1.11.x and "Go Modules".
In a nutshell, Go modules allow to explicitly declare what are the dependencies a given module needs for the go build tool to successfully build it.
And, more importantly, Go modules allow to explicitly declare what are the needed versions of these dependencies, essentially making a build completely reproducible.

You can find more informations about Go modules over there:

Currently, modules are only tested in Travis-CI, on the Go master branch.
But as Go-1.12.x will get closer and modules get more ubiquitous, we'll gradually switch to "Go modules" being the mainstream way to build Go-HEP.

Do not hesitate to report any issues you encounter when building with GO111MODULE=on enabled.

rootio

Another big news for the v0.15.0 release is the support for writing ROOT files:

writing TObjStrings, TH1x, TH2x, TGraph, TGraph{,Assymm}Errors is in,
support for writing compressed ROOT files as well (including lz4, lzma and zlib)
2 new ROOT-related commands:
- cmd/yoda2root: a command to convert YODA files into ROOT ones (so: histograms and scatters)
- rootio/cmd/root-cp: a command to extract objects from a ROOT file into a new ROOT file

To support writing TH1x, TH2x and TGraphs, hbook types have been modified to export most of their fields -- so one can create a rootio.H1D from a hbook.H1D.
This enabled hbook/rootcnv to gain 3 new functions:

rootcnv.FromH1D: a function that converts an hbook.H1D into a rootio.H1D, loosing a bit of informations along the way (ROOT isn't as precise as hbook or YODA are)
rootcnv.FromH2D: a function that converts hbook.H2Ds into rootio.H2Ds,
rootcnv.FromS2D: a function that converts hbook.S2D into rootio.TGraphAsymmErrors.

rootio & xrootd

Finally, we have received 2 patches from Paul Seyfert (a.k.a pseyfert), our first "CERNois" committer :).
Paul enhanced the UI of root-ls and xrd-ls to better deal with nested directories (and how they are displayed) in both of these commands.
Thanks Paul!

Examples

Without further ado, here is how you would create a ROOT file, with lz4 compression, containing a TObjString:

func main() {
	w, err := rootio.Create("out.root", rootio.WithLZ4(flate.BestCompression))
	if err != nil {
		log.Fatal(err)
	}
	defer w.Close()

	var (
		k = "my-objstring"
		v = rootio.NewObjString("Hello World from Go-HEP!")
	)

	err = w.Put(k, v)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("wkeys: %d\n", len(w.Keys()))

	err = w.Close()
	if err != nil {
		log.Fatalf("could not close file: %v", err)
	}
}

and here is how you would use root-cp:

$> root-cp -h
Usage: root-cp [options] file1.root [file2.root [...]] out.root

ex:
 $> root-cp f.root out.root
 $> root-cp f1.root f2.root f3.root out.root
 $> root-cp f1.root:hist* f2.root:h2 out.root

options:

$> root-cp ./testdata/graphs.root:g* out.root
$> root-cp root://xrootd.example.org/file.root:hist.* out.root

More ROOT files writing examples can be found here:

AOB

We will try to have preliminary support for writing TTrees in the next release.
That should be fun.

Interoperability with Apache Arrow Arrays is still on the table.
It might even prove to be easier to support Apache Arrow first and then implement TTrees writing support on top of that.
We will see...

Stay tuned! (and, as always, any kind of help (reviews, patches, nice emails, constructive criticism) deeply appreciated.)

Assets 2

23 Aug 16:16

sbinet

v0.14.0

8996f15

Release v0.14.0

Release v0.14.0 is fresh from the oven.

This release is the result of some massive work in the xrootd package thanks to Mikhail Ivchenko (a.k.a @EgorMatirov), our Google Summer of Code 2018 student.

While GSoC-2018 is now over, it's time to reflect on what Mikhail wrote:

an almost complete xrootd client, compatible with the C++ implementation;
xrd-fuse, a command to mount the contents of a remote XRootD server, locally;
the beginnings of an xrootd server.

Most notably, the client package allowed to:

to create the xrd-cp and xrd-ls commands that copy and list the contents of a remote XRootD server,
to seamlessly read ROOT files over XRootD.

The client package handles authentication with the unix and kerberos protocols.
Unfortunately, authentication via GSI couldn't be implemented because there were no publicly available specifications for that protocol, see xrootd/xrootd#757 for more details.

Here is the final report of this year's GSoC:

https://gist.github.com/EgorMatirov/303e4019606677d1838677a5a7145420

Thanks a lot Mikhail, hope we'll see you around :)

Another big new feature is the ability to write ROOT files, directly with go-hep/rootio.
This is still very much a work in progress, though, as only writing "empty" ROOT files or writing ROOT files with TObjStrings have been explicitly tested.
Next release should see explicit support for writing histograms and graphs.

Lastly, improvements on the build and continuous integration procedure have been applied during this release cycle:

added AppVeyor CI, for Windows:
added code coverage, with codecov.io:

Code coverage improvements

The following packages have been updated, with additional tests, to improve their code coverage:

brio, csvutil, csvutil/csvdriver,
fit, fmom,
heppdt, hepmc, hepevt,
hbook/ntup, hplot,
lcio, lhef,
rootio, sio,
xrootd.

Still some more work is needed to bring code coverage to a saner level (from ~55% to ~70-80%.)
Help more than welcome: it's "just" a matter of creating examples and tests.

hepevt

AsciiEncoder was renamed into simply Encoder
AsciiDecoder was renamed into simply Decoder

rootio

As noted above, it is now possible to create ROOT files.
The rootio package has a couple of examples:

Here is how you would create a ROOT file with one TObjString in it:

package main

import (
	"fmt"
	"log"

	"go-hep.org/x/hep/rootio"
)

func main() {
	w, err := rootio.Create(fname)
	if err != nil {
	    log.Fatal(err)
	}
	defer w.Close()

	var (
	    k   = "my-objstring"
	    v   = rootio.NewObjString("Hello World from Go-HEP!")
	)

	err = w.Put(k, v)
	if err != nil {
	    log.Fatal(err)
	}

	fmt.Printf("wkeys: %d\n", len(w.Keys()))
	
	err = w.Close()
	if err != nil {
	    log.Fatalf("could not close file: %v", err)
	}
}

It is quite possible the API may change as we gain experience with what it ought to look like in a Go world.
e.g. it is possible that rootio.Directory.Put would get clever enough to automatically translate a Go builtin, like string, in the ROOT equivalent, say TObjString, on the fly.
Still pondering on that...

Again, still a lot of work to do on the writing side of things:

support for TH1x,
support for TH2x,
support for TGraph, TGraphErrors, TGraphAsymErrors
support for TTree (this will take some time)
support for user-provided types.

xrootd

The xrootd/server package ships with support for the following XRootD requests:

open, read, close,
write, stat, truncate, sync, rename.

There is no authentication support (yet) on the server: DO NOT RUN THIS ON PUBLICLY ACCESSIBLE MACHINES :)

The xrootd package gained a new sub-command:

xrootd/cmd/xrd-srv

$> xrd-srv -h
xrd-srv serves data from a local filesystem over the XRootD protocol. 

Usage:

 $> xrd-srv [OPTIONS] <base-dir>

Example:

 $> xrd-srv /tmp
 $> xrd-srv -addr=0.0.0.0:1094 /tmp

Options:
  -addr string
    	listen to the provided address (default "0.0.0.0:1094")

AOB

Support for Go-1.6 and Go-1.7 has been dropped.
Please upgrade to the latest and finest Go version (1.11 is around the corner, with support for Go modules.)

Another interesting possible development avenue: exposing ROOT TTrees as Apache Arrow Arrays.
This would allow for a better interoperability with that ecosystem and the tools it provides for data science and data analysis in general.
Help wanted!

Assets 2

Releases: go-hep/hep

Release v0.23.0: root2fits & fits2root

groot

hplot

xrootd

ChangeLog

Release v0.22.0: root-merge and Happy New Year

groot

hbook

FUSE

AOB

ChangeLog

Release v0.21.0: writing trees -- with slices and from structs

groot

hplot

clean-up

AOB

ChangeLog

Release v0.20.0: writing trees

groot

hplot

AOB

Release v0.19.0

groot

AOB

Release v0.18.0

fit

groot

hbook

hep

hplot

xrootd

AOB

Release v0.17.1

brio

groot

xrootd

AOB

Release v0.16.0

groot & rootio

groot/rsrv & root-srv

hbook

hplot

rio

xrootd

AOB

Release v0.15.0

rootio

rootio & xrootd

Examples

AOB

Release v0.14.0

Code coverage improvements

hepevt

rootio

xrootd

AOB