LISA Conference 2004 Notes

http://www.usenix.org/lisa04/ http://lisa04.conference.usenix.org/cgi-bin/lisa04.pl - LISA Wiki

Tutorial Programs

• Sunday November 14, 2004
• S2-System Monitoring and Network Monitoring
  • John Sellens, jsellens@syonexPLEASENOSPAM.com
  • Monitoring - What, Why and Where
    • Don't collect too much
  • Alternatives
  • SNMP Basics

snmpwalk -v 1 localhost -c public system

• • Basic Tools * Standardize on Net-SNMP with sub-agents for Solaris snmp page 73 example
  • Implementation and Challenges
  • Some Monitoring Packages * Nagios is what we want to use...nothing else.
  • Monitoring Devices
  • Application Instrumentation
  • Putting It All Together
  • A Short Simple Case Study
  • Build Your Own
  • Resources
  • Wrap Up

• Monday November 15, 2004
  • M2, System and Network Monitoring Tools in Depth
  • John Sellens, jsellens@syonexPLEASENOSPAM.com
  • Components
  • Nagios
    • Setup a local copy of bind that is a blind authoratitive copy that you get zone transfers and listen only on the loopback. You can then put hostnames and get all current DNS names. address hostname
  • MRTG
  • Cricket
  • Orca
    • orcallator for solaris boxes
    • procallator for linux boxes
  • References

• Tuesday November 16, 2004
  • T10 - Documentation Techniques for System Administrators * Mike Ciavarella, Univ of Melbourne, mikec+lisa2004@csPLEASENOSPAM.mu.oz.au
  • Why Document? * We forget...managers expect it...others need your expertise...accidents
  • Documentation is for someone else, management colleagues users marketing YOU (at 3AM)
  • Audience * State who your docuemnt is for
  • Picture and explanitory text
  • Saves time in the long run, reduces your stress
  • Need to document as you go, COMMENTS HELP
  • CGI - script to add to documents
  • Put it as a part of their job performance review
  • local IRC server?
  • Documentation afternoon - pay for pizza, coke, etc. - once a month, little time commitment
  • Define structure - Templates
  • Define scope - You can negate here (not to tell you this, only this)
  • Define audience
  • Define lifetime/Intended use
  • Sytlesheets are helpful in making it pretty
  • Make it readable - no jargon, no TLA's
  • Include graphs, pictures from start to help with doc from the get go if it is a draft
  • Keep drafts accessible
  • Well structured documents
  • Document overload - split them up
  • Spellchecker - aspell for unix - what about twiki?

• Cover
  • Title
  • Author
  • Revision History
  • Abstract
  • Copyright, disclaimers
• Scope
  • What does this cover, not cover, audience
• Executive Summary
  • Single page - Management
• Background Material
  • Bring reader up to speed, motivations
• Content
  • Treat like a court casee, can't refer to what hasn't been presented.
  • Document sources
• Conclusion & Recommendations
  • Short and targeted
• Tools
  • Visio
• How do we manage our documents?
  • Track revisions, access control, searching, produce offline media(burn cd), notes and evaluations(user feedback), publishing
  • Differing local standards are a problem, ArborText Epic (Not free) * RTFM - Request Tracker * Zope + Silva, Zope + Plone

BOF - Config Management

• pconfig - Argon
• cfengine
• radmind - U of Mich. - Profiles are sets of files
• Tivoli
• opswear
• Veio
• BigFix
• BladeLogic
• Mvalent
• Miramba/BMC
• Webmin
• SMS
• AD

Technical Sessions

• Wednesday November
  • NFS -
    • 2049 is the only port - firewall friendly
    • Solaris 10, AIX 5.3
    • iSCSI - not going to replace FC, but cheaper hardware costs, maybe not as good performance
• Thursday November 18, 2004
  • Flight Data Recorder (FDR)
    • Chad Verbowski, Microsoft Research
    • Auditiing Scenarios
      • Automated Regulatory Change/Config Auditing
      • Trouble Shooting
      • Challenges - Completeness and correctness
    • Configuration Transaction Scenarios
      • Rollback
      • Challenges - Apps not complete, cross application, accuracy
    • What If Scenarios
      • What might be impacted
      • Challenges - Remote calls
    • Benefits
      • Cost, Time
    • Today's Challenges
    • FDR Goals
      • Black Box / non-participatory
      • User Friendly grouping
  • Real-Time Log File Analysis Using the Simple Event Correlator (SEC)
    • John Rouillard, rouilj@csPLEASENOSPAM.umb.edu
    • Who cares about log events? - Nobody cares about events
    • Why not log analysis
      • Too much information, difficult patterns, need to respond to problems indicated by logs
    • How is info in logs presented?
      • Data in the event itself
      • Spread across multiple log entries
      • Absense of an event
      • Relationships between events - failure to success ratio is interesting - trending
    • Multiple events and the tie problem
      • Need to tie or connect events - May be out of order
    • Event relationships
      • Before vs after
      • Sequences - bootup
      • Coincident within window (order unimportant) - event a and event b mus occure in 5 minute to be a problem
      • Reordering issues
    • Detecting missing events
      • Need to know when an event should occure, but didn't
    • Missing event detection: Is the loggng subsystem working?
      • Logger from cron to generate heartbeat every 20 minutes.
    • Is the logging working?

To Do

• logsurfer/SEC/swatch - Which is the best logfile parser?
• Net-SNMP for all platforms to give us a standard base to query against

-- MattMillard - 16 Nov 2004