You are currently browsing the tag archive for the ‘ghc’ tag.

Two months ago Ivan asked me if we had working darcs-2.8 for ghc-7.8 in gentoo. We had a workaround to compile darcs to that day, but darcs did not work reliably. Sometimes it needed 2-3 attempts to pull a repository.

A bit later I’ve decided to actually look at failure case (Issued on darcs bugtracker) and do something about it. My idea to debug the mystery was simple: to reproduce the difference on the same source for ghc-7.6/7.8 and start plugging debug info unless difference I can understand will pop up.

Darcs has great debug-verbose option for most of commands. I used debugMessage function to litter code with more debugging statements unless complete horrible image would emerge.

As you can see in bugtracker issue I posted there various intermediate points of what I thought went wrong (don’t expect those comments to have much sense).

The immediate consequence of a breakage was file overwrite of partially downloaded file. The event timeline looked simple:

  • darcs scheduled for download the same file twice (two jobs in download queue)
  • first download job did finish
  • notified waiter started processing of that downloaded temp file
  • second download started truncating previous complete download
  • notified waiter continued processing partially downloadeed file and detected breakage

Thus first I’ve decided to fix the consequence. It did not fix problems completely, sometimes darcs pull complained about remote repositories still being broken (missing files), but it made errors saner (only remote side was allegedly at fault).

Ideally, that file overwrite should not happen in the first place. Partially, it was temp file predictability.

But, OK. Then i’ve started digging why 7.6/7.8 request download patterns were so severely different. At first I thought of new IO manager being a cause of difference. The paper says it fixed haskell thread scheduling issue (paper is nice even for leisure reading!):

GHC’s RTS had a bug in which yield
placed the thread back on the front of the run queue. This bug
was uncovered by our use of yield
which requires that the thread
be placed at the end of the run queue

Thus I was expecting the bug from this side.

Then being determined to dig A Lot in darcs source code I’ve decided to disable optimizations (-O0) to speedup rebuilds. And, the bug has vanished.

That made the click: unsafePerformIO might be the real problem. I’ve grepped for all unsafePerformIO instances and examined all definition sites.

Two were especially interesting:

-- src/Darcs/Util/Global.hs
-- ...
_crcWarningList :: IORef CRCWarningList
_crcWarningList = unsafePerformIO $ newIORef []
{-# NOINLINE _crcWarningList #-}
-- ...
_badSourcesList :: IORef [String]
_badSourcesList = unsafePerformIO $ newIORef []
{- NOINLINE _badSourcesList -}
-- ...

Did you spot the bug?

Thus The Proper Fix was pushed upstream a month ago. Which means ghc is now able to inline things more aggressively (and _badSourcesList were inlined in all user sites, throwing out all update sites).

I don’t know if those newIORef [] can be de-CSEd if types would have the same representation. Ideally the module also needs -fno-cse, or get rid of unsafePerformIO completely :].

(Side thought: top-level global variables in C style are surprisingly non-trivial in "pure" haskell. They are easy to use via peek / poke (in a racy way), but are hard to declare / initialize.)

I had a question wondered how many haskell packages manage to misspell ghc pragma decparations in a way darcs did it. And there still _is_ a few of such offenders:

$ fgrep -R NOINLINE . | grep -v '{-# NOINLINE' | grep '{-'
--
ajhc-0.8.0.10/lib/jhc/Jhc/List.hs:{- NOINLINE filterFB #-}
ajhc-0.8.0.10/lib/jhc/Jhc/List.hs:{- NOINLINE iterateFB #-}
ajhc-0.8.0.10/lib/jhc/Jhc/List.hs:{- NOINLINE mapFB #-}
--
darcs-2.8.4/src/Darcs/Global.hs:{- NOINLINE _badSourcesList -}
darcs-2.8.4/src/Darcs/Global.hs:{- NOINLINE _reachableSourcesList -}
--
dph-lifted-copy-0.7.0.1/Data/Array/Parallel.hs:{- NOINLINE emptyP #-}
--
dph-par-0.5.1.1/Data/Array/Parallel.hs:{- NOINLINE emptyP #-}
--
dph-seq-0.5.1.1/Data/Array/Parallel.hs:{- NOINLINE emptyP #-}
--
freesect-0.8/FreeSectAnnotated.hs:{- # NOINLINE showSSI #-}
freesect-0.8/FreeSectAnnotated.hs:{- # NOINLINE FreeSectAnnotated.showSSI #-}
freesect-0.8/FreeSect.hs:{- # NOINLINE fs_warn_flaw #-}
--
http-proxy-0.0.8/Network/HTTP/Proxy/ReadInt.hs:{- NOINLINE readInt64MH #-}
http-proxy-0.0.8/Network/HTTP/Proxy/ReadInt.hs:{- NOINLINE mhDigitToInt #-}
--
lhc-0.10/lib/base/src/GHC/PArr.hs:{- NOINLINE emptyP #-}
--
property-list-0.1.0.2/src/Data/PropertyList/Binary/Float.hs:{- NOINLINE doubleToWord64 -}
property-list-0.1.0.2/src/Data/PropertyList/Binary/Float.hs:{- NOINLINE word64ToDouble -}
property-list-0.1.0.2/src/Data/PropertyList/Binary/Float.hs:{- NOINLINE floatToWord32 -}
property-list-0.1.0.2/src/Data/PropertyList/Binary/Float.hs:{- NOINLINE word32ToFloat -}
--
warp-2.0.3.3/Network/Wai/Handler/Warp/ReadInt.hs:{- NOINLINE readInt64MH #-}
warp-2.0.3.3/Network/Wai/Handler/Warp/ReadInt.hs:{- NOINLINE mhDigitToInt #-}

Looks like there is yet something to fix :]

Would be great if hlint would be able to detect pragma-like comments and warn when comment contents is a valid pragma, but comment brackets don’t allow it to fire.

{- NOINLINE foo -} -- bad
{- NOINLINE foo #-} -- bad
{-# NOINLINE foo -} -- bad
{-# NOINLINE foo #-} -- ok

Thanks for reading!

A week ago Austin Seipp and GHC team announced first release candidate from 7.8 branch.

As a packager I was especially interested in following features:

  1. GHCi (and dynamic linking) on unregisterised arches, like ia64 and powerpc64
  2. jobs argument for ghc make. Parallel builds for free.
  3. what did seriously break, what was fixed?

First off, -rc1 is packaged in gentoo-haskell overlay (not keyworded as quite a bit of packages fail to build against ghc-7.8).

GHCi (and dynamic linking)

Dynamic linking works like a charm! GHCi loads binaries noticeaby faster. Let’s test it! Simplest synthetic test: how fast do you get prompt from interpreter?

# ghc-7.6:
$ time { echo '1+1' | ghci -package yesod-core >/dev/null; }
real    0m0.626s
user    0m0.550s
sys     0m0.074s
# ghc-7.8:
$ time { echo '1+1' | ghci -package yesod-core >/dev/null; }
real    0m0.209s
user    0m0.172s
sys     0m0.034s

It’s a case, when files are cached in RAM. 3-4 times faster. The same boost should apply every time when you compile something template-haskell related.

jobs argument for ghc make

I’ve went ahead and tried to enable it for all ebuilds.

For some reason ghc eats a lot of system time in that mode. Likely jobs without arguments is not very good idea and i’ll need to limit it by minimum of MAKEOPTS value and some N (Cabal picked 64).

Even in this mode 2-time speedup is visible on large packages.

So what did break?

Not _that_ much, actually.

alex and happy generated parsers

All package maintainers who ship lexers generated by alex and parsers generated by happy are strongly advised to update those tools locally and reissue hackage update, as old parsers do not compile against ghc-7.8.

If you have happened to use low-level

(==#) :: Int# -> Int# -> Bool

primitives, you might need to port your code a bit, as how their type is a bit different:

(==#) :: Int# -> Int# -> Int#

Here is our example fix for arithmoi.

Type inference changed a bit.

Traditionally darcs needed a patch :] In that big mostly dumb patch most interesting bit is explicit assignment:

- where copyRepo =
+ where copyRepo :: IO ()
+ copyRepo =

Even more amusing breakage was in shake, where error was in inability to infer Addr# argument. No idea was it a bug or feature.

Unsafe removals

As we’ve seen in darcs patch many unsafe${something} functions went away from Foreign modules down to their Unsafe counterparts.

Typeable

Typeable representation did change in a substantial way, thus advanced generic stuff will break. I have no example fix, but have a few of broken packages, like dependent-sum.

Hashtable gone from base

Example of fix for frag package. By the way, ghc-7.6 used to eat 8GBs of RAM compiling frag. For ghc-7.8 it was enough 700MBs even for 8 building threads.

Compiler itself

The thing I expected to try didn’t compile: unregisterised arches and GHCi on them.

I’ve hacked-up a workaround to make them build, but in threaded RTS mode it still SIGSEGVs.

STG gurus are welcome to help me :]

I have fundamental questions like:

  • can unregisterised builds support SMP in theory? (via __thread attribute for example)
  • did UNREG ever produce working threaded runtime?
$ cat __foo/foo.hs 
main = print 1
# non-threaded works, as always been
$ inplace/bin/ghc-stage1 --make __foo/foo.hs -threaded -debug -fforce-recomp
#
$ gdb --args ./__foo/foo +RTS -D{s,i,w,g,G,b,S,t,p,a,l,m,z,c,r}
...
(gdb) run
...
7ffff7fb9700: resuming capability 0
7ffff7fb9700: cap 0: created thread 1
7ffff7fb9700: new bound thread (1)
7ffff7fb9700: cap 0: schedule()
7ffff7fb9700: cap 0: running thread 1 (ThreadRunGHC)
Jumping to 0x7ec17f
#
Program received signal SIGSEGV, Segmentation fault.
0x00000000007ec1a2 in stg_returnToStackTop ()
(gdb) bt
#0  0x00000000007ec1a2 in stg_returnToStackTop ()
#1  0x00000000007d26d9 in StgRun (f=0x7ec17f , basereg=0xca0648) at rts/StgCRun.c:81
#2  0x00000000007c7a30 in schedule (initialCapability=0xca0630, task=0xcc3b30) at rts/Schedule.c:463
#3  0x00000000007ca2c4 in scheduleWaitThread (tso=0x7ffff6b05390, ret=0x0, pcap=0x7fffffffd218) at rts/Schedule.c:2346
#4  0x00000000007c0162 in rts_evalIO (cap=0x7fffffffd218, p=0xb61450 , ret=0x0) at rts/RtsAPI.c:459
#5  0x00000000007e04c3 in ioManagerStartCap (cap=0x7fffffffd218) at rts/posix/Signals.c:184
#6  0x00000000007e04f6 in ioManagerStart () at rts/posix/Signals.c:194
#7  0x00000000007d1d5d in hs_init_ghc (argc=0xc96570 , argv=0xc96578 , rts_config=...) at rts/RtsStartup.c:262
#8  0x00000000007d000b in real_main () at rts/RtsMain.c:47
#9  0x00000000007d0122 in hs_main (argc=17, argv=0x7fffffffd418, main_closure=0xb527a0 , rts_config=...) at rts/RtsMain.c:114
#10 0x0000000000404df1 in main ()

Looks like CurrentTSO is complete garbage. Should not happen :]

Conclusion

The experience is positive. I already get bored, when see single-threaded make of ghc-7.6 and want to update a compiler.

Things like yesod, darcs, hoogle, pandoc and xmonad build fine, thus you can get working environment very fast.

Package authors are more eager to fix stuff for this release: it turns bug lookup and benchmarking into very interactive process.

I want to thank All Of You to make push haskell forward!

Thank you!

Time to time internetz stumble upon a ghci bug which is seen as inability to load base haskell package:

Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... ghc: /usr/lib64/ghc-7.4.2/base-4.5.1.0/HSbase-4.5.1.0.o: unknown symbol `stat'
ghc: unable to load package `base'

But the bug was most popular across rare gentoo users. An interesting correlation!

This post is about the root of this problem: the gory implementation details of ghci dynamic loader down to libC and even ELF symbols!

Not scared? Fasten your belts and Read On!

Read the rest of this entry »

Ladies and gentlemen!

If you happen to be involved in using/developing haskell-powered software you might like to answer our poll on that matter.

Thanks in advance!

Another small breakthrough today for those who would like to see haskell programs running.

Here is a small incomplete HOWTO for gentoo users on how to build a crosscompiler running on x86_64 host targeted on ia64 platform.

It is just an example. You can pick any target.

Read the rest of this entry »

Today I have unmasked ghc-7.6.1 in gentoo‘s haskell overlay. Quite a few of things is broken (like unbumped yet gtk2hs), but major things (like darcs) seem to work fine. Feel free to drop a line on #gentoo-haskell to get the thing fixed.

Some notes and events in the overlay:

  • ghc-7.6.1 is available for all major arches we try to support
  • a few ebuilds of overlay were converted to EAPI=5 to use subslot depends (see below)
  • we’ve got working ghc-9999 ebuild with shared libraries by default! (see below)

Read the rest of this entry »

A year has passed since last update from gentoo haskell team. Quite a few things have happened in haskell land since:

  • ghc-7.2 and 7.4 were released by GHC HQ
  • hackage got ever increasing ton of package bumps
  • overlay got some of those updates
  • cabal test suite by Thomas Tuegel became more popular
  • and many others!

Read the rest of this entry »

Follow

Get every new post delivered to your Inbox.