Secret internals
 - Data
    + how to dig bug data (the btsutils experience)
      BTS source at http://bugs.debian.org/debbugs-source/mainline/
      BTS data at /org/bugs.debian.org/spool/ on merkel
      BTS data rsyncable at merkel.debian.org::bts-spool-db/
        Dir structure:
	  user/  usertags
	  archive/  archived bugs (first dir is the last 2 digits of the bug number)
	  db-h/  active bugs (first dir is the last 2 digits of the bug number)
	Files:
	  *.log     all messages that came in and other BTS annotations
	  *.report  the mail that opened the bug
	  *.status  I don't know
	  *.summary some summary bug information
	Then there are various general data files in spool/ (still undocumented)
      LDAP query
        ldapsearch -p 10101 -h bts2ldap.debian.net -x -b dc=current,dc=bugs,dc=debian,dc=org "(&(debbugsSourcePackage=$SRCPKG)(debbugsState=open))" debbugsID | grep ^debbugsID | sed 's/^debbugsID: //'
      SOAP interfaces to the bts
        http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=377520
	then ask dondelelcaro

      Example code: ~/dev/deb/bts-utils/

    + how to dig control data (the mole experience)
      http://wiki.debian.org/Mole
      merkel:/org/qa.debian.org/mole/db/

	desktopfiles.moledb		.desktop files found in the packages
	dscfiles-control.moledb		all debian/control files
	  Undocumented:
	dscfiles-watch.moledb
	lintian-1.23.22.moledb
	lintian-1.23.24.moledb
	packages-debian-experimental-bin.moledb
	packages-debian-experimental-src.moledb
	packages-debian-oldstable-bin.moledb
	packages-debian-oldstable-src.moledb
	packages-debian-stable-bin.moledb
	packages-debian-stable-src.moledb
	packages-debian-stable_proposed_updates-bin.moledb
	packages-debian-stable_proposed_updates-src.moledb
	packages-debian-testing-bin.moledb
	packages-debian-testing-src.moledb
	packages-debian-testing_proposed_updates-bin.moledb
	packages-debian-testing_proposed_updates-src.moledb
	packages-debian-unstable-bin.moledb
	packages-debian-unstable-src.moledb
	packages-debian_backports-sarge_backports-bin.moledb
	packages-debian_backports-sarge_backports-src.moledb
	packages-debian_security-oldstable_updates-bin.moledb
	packages-debian_security-oldstable_updates-src.moledb
	packages-debian_security-stable_updates-bin.moledb
	packages-debian_security-stable_updates-src.moledb
	packages-debian_security-testing_updates-bin.moledb
	packages-debian_security-testing_updates-src.moledb
	packages-debian_volatile-stable_proposed_updates_volatile-bin.moledb
	packages-debian_volatile-stable_proposed_updates_volatile-src.moledb
	packages-debian_volatile-stable_volatile-bin.moledb
	packages-debian_volatile-stable_volatile-src.moledb
	packages-debian_volatile-stable_volatile_sloppy-bin.moledb
	packages-debian_volatile-stable_volatile_sloppy-src.moledb

      Example code: my tag extraction tools in merkel:bin/

    + LDAP interface to db.debian.org

      ldapsearch -x -h db.debian.org -b dc=debian,dc=org $*

      Example code:
      # Count developers:
      #   bin/ldapsearch '(&(keyfingerprint=*)(gidnumber=800))' | grep ^uid:|wc
      # Stats by nationality:
      #   ldapsearch -h db.debian.org -x -b ou=users,dc=debian,dc=org c | grep ^c: | sort | uniq -c | sort -n | tail

    + The maint/comaint file
      http://qa.debian.org/data/ddpo/results/ddpo_maintainers

    + my aggregated package data (all descs for all arches)
      http://people.debian.org/~enrico/AllPackages.gz
      http://people.debian.org/~enrico/AllPackages-nonexperimental.gz
      
      Example code: how to build a Xapian database with them

    + my popcon aggregated frequencies:

      http://people.debian.org/~enrico/popcon-frequencies.gz

      Example:
      how to compute TFIDF scores of the top packages in one's own system

    + my popcon suggestion interface
      http://people.debian.org/~enrico/anapop
        file upload
	/{stats,xposquery,xnegquery}/token

      Example code: the WebIndex class from anapop.py
      Example code: how to download a suggestion list for the local machine
      Example code: pmnew

    + where to get debtags data
       + raw files

	 On system:
         /var/lib/debtags/package-tags (aggregated)
         /var/lib/debtags/vocabulary
	 grep-dctrl example (from the Packages file)
	 tagcoll query examples (reverse)
	 debtags query examples (bool search, smartsearch)
	 Example code: grep ^pkg: /var/lib/debtags/package-tags

	 Online:
	 http://debtags.alioth.debian.org/tags/tags-current.gz
	 http://debtags.alioth.debian.org/tags/vocabulary.gz

       - indexed data (use libept-dev to access it, show example code)

       + the debtagsd CGIs
         http://debtags.alioth.debian.org/cgi-bin/fts/key1[/key2[/...]]
	   Full text search, returns sorted package list
         http://debtags.alioth.debian.org/cgi-bin/stags/key1[/key2[/...]]
	   Smart tag search, returns sorted tag list
         http://debtags.alioth.debian.org/cgi-bin/sugg/pkg
	   Suggest tags for the package, returns sorted tag list
         http://debtags.alioth.debian.org/cgi-bin/unt[/num]
	   Return the list of packages that have no tags, or no more than 'num'
	   tags
         http://debtags.alioth.debian.org/cgi-bin/notrole
	   Return the list of packages that miss role tags
         http://debtags.alioth.debian.org/cgi-bin/nouitoolkit
	   Return the list of packages that miss uitoolkit tags
         http://debtags.alioth.debian.org/cgi-bin/maint/email
	   Package list for a given maintainer
         http://debtags.alioth.debian.org/cgi-bin/pkgs/tag1[/tag2[...]]
	   List of packages that have at least all the given tags
         http://debtags.alioth.debian.org/cgi-bin/pkg/pkgname
	   Information and long description for one package
         http://debtags.alioth.debian.org/cgi-bin/qstats
	   Quick tag database statistics
         http://debtags.alioth.debian.org/cgi-bin/patch
	   POST request:
	     patch = uri encode of the tag patch
	     tag = token identifying the submitter for helping me to correlate
	           submissions

	 http://debtags.alioth.debian.org/js/debtags.js
	   Functions to easily interface to all these CGIs
	 http://debtags.alioth.debian.org/js/vocabulary.js
	   Javascript-accessible facet and tag descriptions

    - implementing hooks from debian-changes mails

      TODO

    - Debian weather and other EDOS data

      TODO

 - Algorithms
    - dependency resolvers
      germinate
      apt-get showpkg (?)
      others?

    - distro builders
    - EDOS advanced query tools
    - Debtags smart keyword -> tag mappings
 - Libraries for quick development
    - python-debian
    - ?
 - Get the audience to suggest more

19:54 #debian-devel < enrico> Yoe: go for 'Secret debian internals', as a workshop, then
19:55 #debian-devel < enrico> Yoe: I'll show what little goodies I found or
created here and there during my Debian fun time after I unsubscribed from many
debian lists, and ask Debian people in the audience if they have any to add to
the list





17:42 #debian-devel < glatzor> enrico: what is mole?
17:42 #debian-devel < enrico> glatzor: jvw should be able to quickly tell you where to find them
17:42 #debian-devel < enrico> glatzor: link coming
17:42 #debian-devel < GyrosGeier> noshadow, the problem is that I have a broken gateway that claims to be a
                                  nameserver and tells me to recheck via DHCP
17:42 #debian-devel < enrico> glatzor: http://wiki.debian.org/Mole
17:43 #debian-devel < enrico> glatzor: it's a big data mining thing that runs on the archive
17:43 #debian-devel < pusling> deathmole
17:43 #debian-devel < enrico> glatzor: it's collecting .desktop files but I don't remember where
17:43 #debian-devel < ilmari> GyrosGeier: just comment out the make_resolv_conf call in
                              /etc/dhcp3/dhclient-script
17:43 #debian-devel < enrico> glatzor: I remember: merkel:/org/qa.debian.org/mole/db/desktopfiles.moledb
17:44 #debian-devel < glatzor> noshadow: /etc/dhcp3/dhclient.conf
17:44 #debian-devel < ilmari> or maybe just removing the dns-related options from the request statement in
                              dhclient.conf helps
17:44 #debian-devel < GyrosGeier> ilmari, thanks
17:45 #debian-devel < noshadow> glatzgor: ???
17:45 #debian-devel < Manoj> madduck: We can do a joint effort. The idea would be to identify common
                             components in a generic package development workflow, and talk about how each
                             workflow element can be instantiated, and I can come in with the concrete
                             example
17:46 #debian-devel < Manoj> madduck: even identifyying common elements of the routine tasks in developing
                             packages is useful, having at least one detailed set of examples to anchor
                             discussions adds value.
17:49 #debian-devel < enrico> glatzor: in that file you get all the .destop files, indexed by package.  It's
                              a berkeley db file
17:49 #debian-devel < enrico> glatzor: sorry: merkel:/org/qa.debian.org/data/mole/db/desktopfiles.moledb
17:49 #debian-devel < enrico> glatzor: also accessible via http at http://qa.debian.org/data/mole/db/

