A week ago Austin Seipp and GHC team announced first release candidate from 7.8 branch.

As a packager I was especially interested in following features:

  1. GHCi (and dynamic linking) on unregisterised arches, like ia64 and powerpc64
  2. jobs argument for ghc make. Parallel builds for free.
  3. what did seriously break, what was fixed?

First off, -rc1 is packaged in gentoo-haskell overlay (not keyworded as quite a bit of packages fail to build against ghc-7.8).

GHCi (and dynamic linking)

Dynamic linking works like a charm! GHCi loads binaries noticeaby faster. Let’s test it! Simplest synthetic test: how fast do you get prompt from interpreter?

# ghc-7.6:
$ time { echo '1+1' | ghci -package yesod-core >/dev/null; }
real    0m0.626s
user    0m0.550s
sys     0m0.074s
# ghc-7.8:
$ time { echo '1+1' | ghci -package yesod-core >/dev/null; }
real    0m0.209s
user    0m0.172s
sys     0m0.034s

It’s a case, when files are cached in RAM. 3-4 times faster. The same boost should apply every time when you compile something template-haskell related.

jobs argument for ghc make

I’ve went ahead and tried to enable it for all ebuilds.

For some reason ghc eats a lot of system time in that mode. Likely jobs without arguments is not very good idea and i’ll need to limit it by minimum of MAKEOPTS value and some N (Cabal picked 64).

Even in this mode 2-time speedup is visible on large packages.

So what did break?

Not _that_ much, actually.

alex and happy generated parsers

All package maintainers who ship lexers generated by alex and parsers generated by happy are strongly advised to update those tools locally and reissue hackage update, as old parsers do not compile against ghc-7.8.

If you have happened to use low-level

(==#) :: Int# -> Int# -> Bool

primitives, you might need to port your code a bit, as how their type is a bit different:

(==#) :: Int# -> Int# -> Int#

Here is our example fix for arithmoi.

Type inference changed a bit.

Traditionally darcs needed a patch :] In that big mostly dumb patch most interesting bit is explicit assignment:

- where copyRepo =
+ where copyRepo :: IO ()
+ copyRepo =

Even more amusing breakage was in shake, where error was in inability to infer Addr# argument. No idea was it a bug or feature.

Unsafe removals

As we’ve seen in darcs patch many unsafe${something} functions went away from Foreign modules down to their Unsafe counterparts.

Typeable

Typeable representation did change in a substantial way, thus advanced generic stuff will break. I have no example fix, but have a few of broken packages, like dependent-sum.

Hashtable gone from base

Example of fix for frag package. By the way, ghc-7.6 used to eat 8GBs of RAM compiling frag. For ghc-7.8 it was enough 700MBs even for 8 building threads.

Compiler itself

The thing I expected to try didn’t compile: unregisterised arches and GHCi on them.

I’ve hacked-up a workaround to make them build, but in threaded RTS mode it still SIGSEGVs.

STG gurus are welcome to help me :]

I have fundamental questions like:

  • can unregisterised builds support SMP in theory? (via __thread attribute for example)
  • did UNREG ever produce working threaded runtime?
$ cat __foo/foo.hs 
main = print 1
# non-threaded works, as always been
$ inplace/bin/ghc-stage1 --make __foo/foo.hs -threaded -debug -fforce-recomp
#
$ gdb --args ./__foo/foo +RTS -D{s,i,w,g,G,b,S,t,p,a,l,m,z,c,r}
...
(gdb) run
...
7ffff7fb9700: resuming capability 0
7ffff7fb9700: cap 0: created thread 1
7ffff7fb9700: new bound thread (1)
7ffff7fb9700: cap 0: schedule()
7ffff7fb9700: cap 0: running thread 1 (ThreadRunGHC)
Jumping to 0x7ec17f
#
Program received signal SIGSEGV, Segmentation fault.
0x00000000007ec1a2 in stg_returnToStackTop ()
(gdb) bt
#0  0x00000000007ec1a2 in stg_returnToStackTop ()
#1  0x00000000007d26d9 in StgRun (f=0x7ec17f , basereg=0xca0648) at rts/StgCRun.c:81
#2  0x00000000007c7a30 in schedule (initialCapability=0xca0630, task=0xcc3b30) at rts/Schedule.c:463
#3  0x00000000007ca2c4 in scheduleWaitThread (tso=0x7ffff6b05390, ret=0x0, pcap=0x7fffffffd218) at rts/Schedule.c:2346
#4  0x00000000007c0162 in rts_evalIO (cap=0x7fffffffd218, p=0xb61450 , ret=0x0) at rts/RtsAPI.c:459
#5  0x00000000007e04c3 in ioManagerStartCap (cap=0x7fffffffd218) at rts/posix/Signals.c:184
#6  0x00000000007e04f6 in ioManagerStart () at rts/posix/Signals.c:194
#7  0x00000000007d1d5d in hs_init_ghc (argc=0xc96570 , argv=0xc96578 , rts_config=...) at rts/RtsStartup.c:262
#8  0x00000000007d000b in real_main () at rts/RtsMain.c:47
#9  0x00000000007d0122 in hs_main (argc=17, argv=0x7fffffffd418, main_closure=0xb527a0 , rts_config=...) at rts/RtsMain.c:114
#10 0x0000000000404df1 in main ()

Looks like CurrentTSO is complete garbage. Should not happen :]

Conclusion

The experience is positive. I already get bored, when see single-threaded make of ghc-7.6 and want to update a compiler.

Things like yesod, darcs, hoogle, pandoc and xmonad build fine, thus you can get working environment very fast.

Package authors are more eager to fix stuff for this release: it turns bug lookup and benchmarking into very interactive process.

I want to thank All Of You to make push haskell forward!

Thank you!

Time to time internetz stumble upon a ghci bug which is seen as inability to load base haskell package:

Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... ghc: /usr/lib64/ghc-7.4.2/base-4.5.1.0/HSbase-4.5.1.0.o: unknown symbol `stat'
ghc: unable to load package `base'

But the bug was most popular across rare gentoo users. An interesting correlation!

This post is about the root of this problem: the gory implementation details of ghci dynamic loader down to libC and even ELF symbols!

Not scared? Fasten your belts and Read On!

Read the rest of this entry »

I’d like to ask gentoo-haskell community for help. We have a nice wiki and our project page have moved there. But it seems that we don’t have enough documentation quality for end-user application. As a developers we support proper builds and tests for that packages but we are not expert users for many of them. So I’d like to ask community to add some docs and tips for applications you use. This basically means installation, advanced config (examples), interesting use cases, links to external resources (blog posts/documentation) and so on. It can help a lot for new Gentoo users.

The most interesting projects are:

Thanks!

Ladies and gentlemen!

If you happen to be involved in using/developing haskell-powered software you might like to answer our poll on that matter.

Thanks in advance!

Another small breakthrough today for those who would like to see haskell programs running.

Here is a small incomplete HOWTO for gentoo users on how to build a crosscompiler running on x86_64 host targeted on ia64 platform.

It is just an example. You can pick any target.

Read the rest of this entry »

Today I have unmasked ghc-7.6.1 in gentoo‘s haskell overlay. Quite a few of things is broken (like unbumped yet gtk2hs), but major things (like darcs) seem to work fine. Feel free to drop a line on #gentoo-haskell to get the thing fixed.

Some notes and events in the overlay:

  • ghc-7.6.1 is available for all major arches we try to support
  • a few ebuilds of overlay were converted to EAPI=5 to use subslot depends (see below)
  • we’ve got working ghc-9999 ebuild with shared libraries by default! (see below)

Read the rest of this entry »

A year has passed since last update from gentoo haskell team. Quite a few things have happened in haskell land since:

  • ghc-7.2 and 7.4 were released by GHC HQ
  • hackage got ever increasing ton of package bumps
  • overlay got some of those updates
  • cabal test suite by Thomas Tuegel became more popular
  • and many others!

Read the rest of this entry »

The Gentoo Linux Haskell Project recently moved to using git on github.

We’ve got various repositories, some larger with several contributors and some smaller with only one developer. All repos were based on the darcs revision control system.

You may browse the new repos at our new home.

Overlay repository

The overlay repository is the heart of the Gentoo Linux Haskell Project. The most common packages are available in the portage tree, and thus available to all Gentoo Linux users without any additional configuration on their part.

For all other packages we use the overlay. It can be packages that change rapidly, are tricky to build, etc.

It’s our main repo, ~4000 commits from many users. There was two tools to consider:

We’ve decided to try both.

darcs-to-git

mkdir overlay.git && cd overlay.git
darcs-to-git ../overlay
git commit --allow-empty -m "phony" # hack, described later
darcs-to-git ../overlay

The hack with –allow-empty is used to workaround an error:

Running: ["git", "log", "-n1", "--no-color"]
fatal: bad default revision 'HEAD'

git does not track directory creation commits (when no files are affected). It’s our first commit. To be reported upstream.

darcs-to-git took 7.5 hours(!) to convert our repo.

darcs-fastconvert

mkdir overlay.git && cd overlay.git
(cd ../overlay ; darcs-fastconvert export) | git fast-import

It was very fast! Took less, than 7 minutes to convert everything (~60 times faster than darcs-to-git!)

some notes

  • darcs-fastconvert does not try to make prettier email-only usernames:

    username ‘john@doe’ becomes ‘john@doe <unknown>’. Patch to convert such names to ‘john <john@doe>’ sent upstream (left copy here).

  • darcs-fastconvert does not filter out empty commits (directory-adding in darcs), so in order to get the same amount of commits as for darcs-to-git you will need to run git filter-branch --prune-empty -f

hackport

hackport is the tool we use to generate Gentoo ebuilds from Hackage packages. It geatly simplifies the process and enables us to cover hundreds of packages with relatively few developers.

The hackport project started off as part of the overay repository in 2005. At some point we decided that it deserved its own repository, as it really was a standalone project. The development was forked from the overlay repository, and continued without being mixed with the overlay commits. However, the result was a repository with an messy history: hackport and overlay stuff was mixed. Moving to git gave us a new chance to clean it up.

When using git you have the option of changeing the history of your repository. Of course this is a powerful tool, but it should be used carefully. As we were in a transition of moving to git, we used these advantages. In general git will require greater knowledge of your consequences than similar tools (darcs, mercurial).

For this job we used the features of git filter-branch, see the git documentation at filter-branch. As the projects where clearly separated it was easy to tell git wich files were interesting:

git filter-branch --tree-filter \
      'rm -rf ignore-this-file and-this-directory' HEAD
git filter-branch --prune-empty -f

We repeated until we’ve cleared the history from the overlay commits. The result is clean and only contains the hackport project. You find it at hackport.

This way we could separate the ~400 commits from the ~1100 commits that had nothing to do with the hackport project.

keyword-stat

keyword-stat is a tool to help us see the status of packages regarding Gentoo’s concept of stable and testing status. Each Gentoo user is able to choose the stability level of each package through the keywording concept.

The repo was already at a nice state and the conversion was straight forward:

mkdir keyword-stat.git && cd keyword-stat.git
git init
( cd ../keyword-stat ; darcs-fastconvert export ) \
    | git fast-import

We have moved gentoo-haskell overlay to github!

It will require some additional actions for overlay users:

layman -f
layman -d haskell
layman -a haskell

Our move was stimulated by a couple of events:

  • code.haskell.org was not very reliable, and

  • as you might notice haskell overlay was (and still is) inaccessible since last week. Current status of code.haskell.org can be tracked here.

Good news!

We’ve got ghc-6.12.3 stable on x86, amd64 and sparc arches! (more to come)

What does it mean for end user?

You’ll get latest haskell compiler and will be able to taste it’s new features! We also got rid of hacky ghc-updater and switched to haskell-updater.

Unfortunately, some haskell packages in main tree might stop building on new compiler. Please report them as problematic to gentoo bugzilla or drop a line in #gentoo-haskell on freenode IRC network.

I’ve decided to look back and estimate timeframe it took us to deliver ghc for you:

So, it took us almost 4 months.

The major problems were:

  • Resurrect GHC on exotic architectures (we have 10 patches on top of vanilla ghc!). Unfortunately, hppa support was lost.
  • Fix packages breaking with 6.10/6.12 branch of ghc (tons of them)

Brave souls can try to install ghc-7-rc2 (aka 7.0.0.20101028) from overlay (currently masked). It has no base-3 (deprecated in ghc-6.10), so you’ll have great chance to become a contributor to various haskell projects!

Enjoy!

Follow

Get every new post delivered to your Inbox.