Logs for #nikola for 2015-03-06

02:08:41 <erdgeist> Do I understand correctly that dependencies in nikola are only to detect if certain artefacts need to be rebuilt, but not to determine the order in which posts need to be rendered the first time?
13:40:24 <ralsina> erdgeist: yes, the order once all dependencies are sorted out is not known
14:24:00 <ChrisWarrick> erdgeist: however, a post could depend on another post
14:53:00 -travis-ci:#nikola- getnikola/nikola#4662 (master - 37ebddc : Chris Warrick): The build is still failing.
14:53:01 -travis-ci:#nikola- Change view: https://github.com/getnikola/nikola/compare/7363b868e51b...37ebddc5af0c
14:53:01 -travis-ci:#nikola- Build details: http://travis-ci.org/getnikola/nikola/builds/53243206
15:05:33 -travis-ci:#nikola- getnikola/nikola#4662 (master - 37ebddc : Chris Warrick): The build was fixed.
15:05:34 -travis-ci:#nikola- Change view: https://github.com/getnikola/nikola/compare/7363b868e51b...37ebddc5af0c
15:05:34 -travis-ci:#nikola- Build details: http://travis-ci.org/getnikola/nikola/builds/53243206
18:49:18 <erdgeist> ChrisWarrick: well, for all the index pages the dependencies must be re-evaluated each time a new post is added. It might change the years index and the tags index
18:49:44 <erdgeist> ChrisWarrick: how does doit figure out, that these pages need to be rebuilt?
18:49:58 <ChrisWarrick> erdgeist: index pages depend on post files/pages
18:50:24 <ChrisWarrick> erdgeist: the post files change (not sure if it’s posts/, cache/, or output/) and this triggers a rebuild
18:51:21 <ChrisWarrick> erdgeist: furthermore, the post creation task has the files as their targets, which is also responsible for dependency calculation
18:53:13 <erdgeist> ChrisWarrick: Okay, and pages are different? I only noticed dependecies added AFTER the pages have been rendered
18:53:50 <ChrisWarrick> erdgeist: dependencies are set when tasks are generated
18:55:22 <ChrisWarrick> erdgeist: how did you notice “dependencies added after pages rendered”?
18:55:26 <erdgeist> ChrisWarrick: Maybe I am not making myself clear enough ;) There seems to be a concept of index pages that need to be rendered last
18:55:49 <erdgeist> ChrisWarrick: the .dep-File is only generated after the html
18:56:06 <erdgeist> ChrisWarrick: and then re-read
18:56:13 <ChrisWarrick> erdgeist: yeah, but the .dep files are used only for dependencies from reStructuredText contents
18:56:17 <ChrisWarrick> erdgeist: index pages must be rendered after all cache/ files that exist in this index page
18:56:32 <erdgeist> ChrisWarrick: correct
18:56:58 <erdgeist> ChrisWarrick: And I need that for pages that contain post-list directives
18:57:13 <erdgeist> ChrisWarrick: because they obviously depend on other pages
18:57:26 <ChrisWarrick> erdgeist: they depend on other pages if and only if you use post.text in the template
18:57:43 <erdgeist> ChrisWarrick: nope
18:57:56 <erdgeist> ChrisWarrick: did you get my pastie?
18:58:06 <erdgeist> 22:41 < erdgeist> Now I already get a stack trace from doit when using a post-list-directive: http://pastie.org/10003313
18:58:52 <erdgeist> the template there does not use anything from the post
18:58:55 <ChrisWarrick> erdgeist: what is stories/music/index.rst?
18:59:42 <erdgeist> .. post-list:: :all: :tags: Liedermacher :id: singersongwriter
19:00:00 <ChrisWarrick> and what is your template?
19:00:19 <erdgeist> the default from base
19:02:11 <erdgeist> and the missing Fine.html would be one of the pages aggregated from this post-list
19:02:42 <erdgeist> maybe its a bug in doit, idk
19:03:25 <erdgeist> it fails, because it cant save the dependency's file stamp
19:03:47 <erdgeist> because dependency was calculated but not yet generated
19:05:18 <ChrisWarrick> erdgeist: paste output of  nikola dumpdb
19:06:40 <erdgeist> DBM type is 'dbm'
19:06:41 <erdgeist> ERROR: ndbm does not support iteration of elements
19:07:52 <ChrisWarrick> sheesh
19:08:06 <erdgeist> kurwa
19:08:24 <ChrisWarrick> tell you what, I can’t help you
19:08:36 <ChrisWarrick> I can’t see ANY dependency creation in post_list
19:09:03 <erdgeist> :)
19:09:30 <erdgeist> I would just add a header to posts containing post-lists, so they are de-prioritized
19:10:00 <ChrisWarrick> I would tackle your fancy indexes differently
19:10:16 <ChrisWarrick> Just use regular indexes and stick some extra text on the first page
19:10:31 <erdgeist> let me show you
19:10:56 <erdgeist> ChrisWarrick: https://erdgeist.org/new/
19:11:42 <erdgeist> ChrisWarrick: the list of projects and blog post teasers is generated, and it works reasonably well
19:12:07 <erdgeist> ChrisWarrick: I just need a way to have some pages treated like index pages and rendered last
19:13:01 <ChrisWarrick> You could try doing this by making sure post.fragment_deps() has all the posts displayed enlisted
19:14:38 <ChrisWarrick> this will require a few changes to the nikola source code, it’s not trivial though
19:15:21 <ChrisWarrick> you could modify post_list_directive so that it talks to record_dependencies and hope for the best
19:15:40 <ChrisWarrick> self.state.document.settings.record_dependencies.add(fpath)
19:17:24 <ChrisWarrick> where fpath is an individual file path, should be AT LEAST post.source_path, post.translated_base_path(lang), os.path.join(self.site.config['OUTPUT_FOLDER'], post.destination_path(lang))
19:18:07 <erdgeist> do you know who wrote the post-list directive feature, so I could ask directly?
19:18:42 <ChrisWarrick> @udono in github, Udo Spallek
19:18:45 <ChrisWarrick> on*
19:18:49 <erdgeist> thanks
19:19:29 <ChrisWarrick> good luck with all that!
19:20:19 <erdgeist> thanks a lot. I duly took notes of all your suggestions :)
19:20:51 <ChrisWarrick> dependency recording could do it, actually
19:25:21 <ChrisWarrick> wah
19:25:30 <ChrisWarrick> the actual problem appears when you start recording dependencies
19:31:06 <ChrisWarrick> All this happens because the dependency files are not used in the original generation and you get a nice little race condition
19:31:17 <erdgeist> yep
19:31:50 <ChrisWarrick> actually, there is one hack you could try
19:32:10 <erdgeist> I am just looking into how from post_pages the tasks are generated
19:32:24 <ChrisWarrick> by time of post creation
19:32:56 <erdgeist> Hmm, I could re-sort them
19:32:59 <ChrisWarrick> nah
19:33:10 <ChrisWarrick> just set the date of your index post to 1970-01-01T00:00:00+00:00
19:33:12 <ChrisWarrick> that will be enough
19:34:52 <ChrisWarrick> the timeline is ALWAYS sorted in reverse chronological order, that’s guaranteed
19:35:21 <erdgeist> getting the same errors
19:36:10 <ChrisWarrick> you must have modified some code somewhere
19:36:47 <ChrisWarrick> try nuking the cache and database just in case
19:37:00 <erdgeist> database is .doit.db?
19:38:00 <ChrisWarrick> yes
19:38:41 <ChrisWarrick> I just successfully produced a page with post.text() in the template and no problems
19:39:18 <ChrisWarrick> I modified post-list to register dependencies on the source file (.rst) and it works fine
19:39:53 <ChrisWarrick> And I also get dependencies for the cache/ files from post.text() apparently
19:41:30 <erdgeist> ChrisWarrick: can I just send you the source for my page and you try it out? It's just one MV
19:41:33 <erdgeist> MB
19:41:35 <ChrisWarrick> go on
19:42:53 <erdgeist> https://erdgeist.org/erdgeist.org.tbz
19:45:38 <ChrisWarrick> erdgeist: 20:33:10 <+ChrisWarrick> just set the date of your index post to 1970-01-01T00:00:00+00:00
19:46:21 <erdgeist> -rw-r--r--  1 erdgeist  503  221  1 Jan  1970 stories/poetry/index.rst
19:46:50 <erdgeist> maybe I forgot to use -p on my tar command
19:47:03 <ChrisWarrick> oh
19:47:08 <ChrisWarrick> you set the file system date
19:47:19 <ChrisWarrick> I meant the .. date: meta field of your post
19:47:31 <erdgeist> oh, how embarrassing
19:48:03 <ChrisWarrick> Nikola itself doesn’t care about file dates, they cannot be trusted in most cases
19:48:54 <erdgeist> much more betterer
19:49:05 <erdgeist> thanks!
19:49:21 <ChrisWarrick> again, a lot of things done in this is using hacks
19:49:58 <ChrisWarrick> one day we could sort site.timeline by /dev/urandom and break everything (that’s nonsense, we won’t)
19:50:07 <ChrisWarrick> bonus question: what magic sorcery did you use to set the file creation date?
19:50:47 <erdgeist> touch -t 197001010000 index.rst 
19:50:54 <erdgeist> posix voodoo
19:52:22 <ChrisWarrick> I always forget touch can be used to do this
19:52:48 <ChrisWarrick> in fact, it’s its main function, and creating new files only is not the intended functionality
19:53:24 <erdgeist> hehe
19:53:51 <erdgeist> I usually type "touch .pnwed" on other ppls servers
19:58:52 -GitHub[nikola]:#nikola- [nikola] Kwpolska pushed 2 new commits to master: http://git.io/pf3W
19:58:52 -GitHub[nikola]:#nikola- nikola/master df2eb6a Chris Warrick: record some deps in post_list...
19:58:52 -GitHub[nikola]:#nikola- nikola/master a4ff446 Chris Warrick: fix #1629 -- document different input formats...
20:04:12 -GitHub[nikola]:#nikola- [nikola] Kwpolska pushed 1 new commit to master: http://git.io/pfZU
20:04:12 -GitHub[nikola]:#nikola- nikola/master 41356a1 Chris Warrick: fix #1625 -- ignore post meta in two-file posts...
20:13:32 -travis-ci:#nikola- getnikola/nikola#4663 (master - a4ff446 : Chris Warrick): The build was broken.
20:13:33 -travis-ci:#nikola- Change view: https://github.com/getnikola/nikola/compare/37ebddc5af0c...a4ff4460af79
20:13:33 -travis-ci:#nikola- Build details: http://travis-ci.org/getnikola/nikola/builds/53389001
20:15:44 -travis-ci:#nikola- getnikola/nikola#4664 (master - 41356a1 : Chris Warrick): The build was broken.
20:15:45 -travis-ci:#nikola- Change view: https://github.com/getnikola/nikola/compare/a4ff4460af79...41356a1ee9ad
20:15:45 -travis-ci:#nikola- Build details: http://travis-ci.org/getnikola/nikola/builds/53389441
21:02:31 -GitHub[nikola]:#nikola- [nikola] Kwpolska pushed 1 new commit to master: http://git.io/pfS8
21:02:31 -GitHub[nikola]:#nikola- nikola/master 9762df2 Chris Warrick: flake8...
21:11:03 -travis-ci:#nikola- getnikola/nikola#4665 (master - 9762df2 : Chris Warrick): The build was fixed.
21:11:04 -travis-ci:#nikola- Change view: https://github.com/getnikola/nikola/compare/41356a1ee9ad...9762df25096a
21:11:04 -travis-ci:#nikola- Build details: http://travis-ci.org/getnikola/nikola/builds/53395474